2024-02-19T18:37:54.746 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T18:37:54.746 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631 --name pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose -- /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631/orig.config.yaml 2024-02-19T18:37:54.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T18:37:54.886 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631 branch: wip-batrick-testing-20240215.160715 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7566631' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240215.160715 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: f78a58c0ffd401d1493058a1022c35f011d65275 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f78a58c0ffd401d1493058a1022c35f011d65275 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: i64440 sha1: 1ce7ea8877c709fa862eac608550374bd96e166e owner: scheduled_pdonnell@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2975 sha1: f78a58c0ffd401d1493058a1022c35f011d65275 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: i64440 suite_path: /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 1ce7ea8877c709fa862eac608550374bd96e166e targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPe8hnfRIFRI8wD79N7u+QwiwW+B45WxJm/l+FtnC+lCvKXZUkLj9wLZNhUmYGPv/cKYsr8nvvvV06YhmdzkfHA= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSEW7KE6ueqFDGMgunBxOgX2IxzTCdBN4+AnkrFqCmo183I0NL8nT5Q+i1dipidEB4W7OyHGPOE2puDgLMta3Y= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - full_sequential_finally: - sleep: duration: 3600 - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/pdonnell/src/github.com_batrick_ceph_i64440/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_18:28:45 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T18:37:54.886 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa; will attempt to use it 2024-02-19T18:37:54.887 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks 2024-02-19T18:37:54.887 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T18:37:54.892 INFO:teuthology.task.internal:Checking packages... 2024-02-19T18:37:54.915 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f78a58c0ffd401d1493058a1022c35f011d65275' 2024-02-19T18:37:54.915 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:37:54.915 INFO:teuthology.packaging:ref: None 2024-02-19T18:37:54.915 INFO:teuthology.packaging:tag: None 2024-02-19T18:37:54.915 INFO:teuthology.packaging:branch: wip-batrick-testing-20240215.160715 2024-02-19T18:37:54.915 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:37:54.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240215.160715 2024-02-19T18:37:55.060 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1450.gf78a58c0 2024-02-19T18:37:55.061 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T18:37:55.068 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T18:37:55.068 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T18:37:55.106 INFO:teuthology.task.internal:Saving configuration 2024-02-19T18:37:55.124 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T18:37:55.130 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T18:37:55.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:53.789618', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPe8hnfRIFRI8wD79N7u+QwiwW+B45WxJm/l+FtnC+lCvKXZUkLj9wLZNhUmYGPv/cKYsr8nvvvV06YhmdzkfHA='} 2024-02-19T18:37:55.175 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:53.791271', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSEW7KE6ueqFDGMgunBxOgX2IxzTCdBN4+AnkrFqCmo183I0NL8nT5Q+i1dipidEB4W7OyHGPOE2puDgLMta3Y='} 2024-02-19T18:37:55.175 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T18:37:55.181 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T18:37:55.182 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T18:37:55.182 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T18:37:55.266 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f99e680f0d0>, signals=[15]) 2024-02-19T18:37:55.267 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T18:37:55.273 INFO:teuthology.task.internal:Opening connections... 2024-02-19T18:37:55.273 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T18:37:55.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:37:55.381 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-02-19T18:37:55.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:37:55.469 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T18:37:55.476 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-02-19T18:37:55.493 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-02-19T18:37:55.493 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-02-19T18:37:55.549 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T18:37:55.550 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T18:37:55.551 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-02-19T18:37:55.572 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-02-19T18:37:55.589 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-02-19T18:37:55.589 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-02-19T18:37:55.646 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-02-19T18:37:55.647 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T18:37:55.648 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T18:37:55.648 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T18:37:55.648 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-02-19T18:37:55.672 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T18:37:55.683 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T18:37:55.707 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T18:37:55.707 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-02-19T18:37:55.710 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-02-19T18:37:55.726 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T18:37:55.730 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T18:37:55.731 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-02-19T18:37:55.767 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-02-19T18:37:55.794 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T18:37:55.847 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T18:37:55.863 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T18:37:55.864 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T18:37:55.864 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T18:37:55.865 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T18:37:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:37:55.881 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-02-19T18:37:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:37:55.882 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-02-19T18:37:55.936 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2024-02-19T18:37:55.937 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2024-02-19T18:37:55.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-02-19T18:37:55.939 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-361.el9.x86_64 2024-02-19T18:37:55.939 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.14.0-361.el9.x86_64 2024-02-19T18:37:55.939 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2024-02-19T18:37:57.962 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-19T18:37:58.101 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-19T18:37:59.437 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-02-19T18:37:59.760 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-02-19T18:38:00.536 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-02-19T18:38:10.100 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 832 kB/s | 8.0 MB 00:09 2024-02-19T18:38:19.792 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-02-19T18:38:23.776 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 886 kB/s | 19 MB 00:22 2024-02-19T18:38:26.550 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-02-19T18:38:31.051 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 4.4 kB/s | 15 kB 00:03 2024-02-19T18:38:31.321 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-02-19T18:38:32.835 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-02-19T18:39:18.865 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 454 kB/s | 20 MB 00:45 2024-02-19T18:39:23.932 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-19T18:39:26.230 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T18:39:26.230 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T18:39:26.330 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:39:26.342 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:39:26.342 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:39:26.342 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:39:26.342 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2024-02-19T18:39:26.343 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:39:26.344 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 100 M 2024-02-19T18:39:26.344 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 123 M 2024-02-19T18:39:26.344 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:39:28.783 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 2.5 MB/s | 6.1 MB 00:02 2024-02-19T18:39:29.275 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 7.1 MB/s | 21 MB 00:02 2024-02-19T18:39:32.083 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 6.8 MB/s | 39 MB 00:05 2024-02-19T18:39:32.416 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): kernel-modules-core-5.14.0-421.el9.x86_6 9.2 MB/s | 33 MB 00:03 2024-02-19T18:39:32.420 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:39:32.420 INFO:teuthology.orchestra.run.smithi039.stdout:Total 16 MB/s | 100 MB 00:06 2024-02-19T18:39:32.420 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:39:32.865 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:39:32.865 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:39:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:39:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:39:36.463 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:39:37.142 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:39:37.275 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:39:40.133 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:39:40.319 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:39:45.384 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:39:45.491 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:39:50.577 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:21.179 INFO:teuthology.orchestra.run.smithi039.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T18:40:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:40:21.251 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:21.781 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:21.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:40:21.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:40:21.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:40:22.257 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:40:22.772 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:40:22.919 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:22.919 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:40:22.919 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T18:40:22.920 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T18:40:22.920 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T18:40:22.920 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T18:40:22.920 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T18:40:22.920 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:40:23.034 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:23.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:40:23.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 68.el9 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-02-19T18:40:23.084 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8264254 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:40:23.085 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T18:40:23.086 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T18:40:23.086 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T18:40:23.086 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-02-19T18:40:23.086 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:40:23.086 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T18:40:23.087 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T18:40:23.087 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 5.14.0-421.el9.x86_64 2024-02-19T18:40:23.087 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T18:40:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-02-19T18:40:25.021 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T18:40:25.067 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-02-19T18:40:25.069 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-02-19T18:40:25.084 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.qRNj6BjLRD 2024-02-19T18:40:25.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qRNj6BjLRD 2024-02-19T18:40:25.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.qRNj6BjLRD 2024-02-19T18:40:25.312 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.qRNj6BjLRD is 6KB 2024-02-19T18:40:25.370 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.qRNj6BjLRD 2024-02-19T18:40:25.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T18:40:25.452 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T18:40:25.453 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T18:40:25.453 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T18:40:25.453 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T18:40:25.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T18:40:25.608 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-02-19T18:40:45.721 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 155 kB/s | 20 MB 02:14 2024-02-19T18:40:50.546 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-02-19T18:40:52.703 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T18:40:52.703 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T18:40:52.800 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T18:40:52.806 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:40:52.807 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 100 M 2024-02-19T18:40:52.808 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 123 M 2024-02-19T18:40:52.808 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:40:53.206 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 15 MB/s | 6.1 MB 00:00 2024-02-19T18:40:53.889 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 19 MB/s | 21 MB 00:01 2024-02-19T18:40:54.631 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 24 MB/s | 33 MB 00:01 2024-02-19T18:40:55.323 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-02-19T18:40:55.326 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:40:55.326 INFO:teuthology.orchestra.run.smithi086.stdout:Total 40 MB/s | 100 MB 00:02 2024-02-19T18:40:55.327 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:40:55.612 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T18:40:55.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:40:55.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:40:55.692 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:40:55.692 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:40:56.724 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:40:56.724 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:40:59.162 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:40:59.946 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:41:00.073 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:41:02.769 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:41:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:41:07.705 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:41:07.912 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:41:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:41:30.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-02-19T18:41:39.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:41:39.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:41:41.930 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:41:41.930 INFO:teuthology.orchestra.run.smithi086.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T18:41:41.931 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:41:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:41:42.239 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-02-19T18:41:42.375 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:41:42.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:41:42.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:41:42.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:41:42.826 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:41:42.826 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:41:42.827 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:41:43.314 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:41:43.449 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:41:43.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:41:43.449 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T18:41:43.449 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T18:41:43.450 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T18:41:43.450 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T18:41:43.450 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T18:41:43.450 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:41:43.562 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:41:43.562 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:41:43.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-02-19T18:41:43.606 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T18:41:43.607 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:41:43.608 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T18:41:43.613 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T18:41:43.613 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-421.el9.x86_64 2024-02-19T18:41:43.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T18:41:44.189 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-02-19T18:41:45.584 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T18:41:45.632 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-02-19T18:41:45.633 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-02-19T18:41:45.648 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.IoBYepTv4A 2024-02-19T18:41:45.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IoBYepTv4A 2024-02-19T18:41:45.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.IoBYepTv4A 2024-02-19T18:41:45.855 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.IoBYepTv4A is 6KB 2024-02-19T18:41:45.906 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.IoBYepTv4A 2024-02-19T18:41:45.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T18:41:45.985 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T18:41:45.985 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T18:41:45.985 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T18:41:45.985 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T18:41:45.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T18:41:46.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-02-19T18:41:54.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:41:54.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:42:16.151 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T18:42:16.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T18:42:16.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:42:50.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T18:42:54.255 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T18:42:59.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T18:42:59.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:43:09.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:43:09.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:43:09.675 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-02-19T18:43:10.049 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:43:10.049 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-421.el9.x86_64"... 2024-02-19T18:43:10.050 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-02-19T18:43:10.068 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-421.el9.x86_64 2024-02-19T18:43:10.069 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T18:43:10.069 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T18:43:10.069 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T18:43:11.070 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T18:43:11.071 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T18:43:11.147 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-02-19T18:43:11.177 DEBUG:teuthology.parallel:result is None 2024-02-19T18:43:18.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T18:43:30.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T18:43:30.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:43:30.826 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-02-19T18:43:31.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T18:43:31.302 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-421.el9.x86_64"... 2024-02-19T18:43:31.302 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-02-19T18:43:31.317 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-421.el9.x86_64 2024-02-19T18:43:31.317 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T18:43:31.317 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T18:43:31.317 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T18:43:32.318 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T18:43:32.318 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T18:43:32.378 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-02-19T18:43:32.416 DEBUG:teuthology.parallel:result is None 2024-02-19T18:43:32.417 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T18:43:32.427 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T18:43:32.427 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T18:43:32.430 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T18:43:32.449 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T18:43:32.475 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T18:43:32.505 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T18:43:32.506 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T18:43:32.508 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T18:43:32.546 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T18:43:32.565 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T18:43:32.566 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T18:43:32.586 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T18:43:32.628 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:43:32.631 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:43:32.641 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:43:32.649 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:43:32.649 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T18:43:32.667 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T18:43:32.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T18:43:32.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T18:43:32.732 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T18:43:32.778 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T18:43:32.778 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T18:43:32.789 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T18:43:32.814 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:43:32.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:43:32.961 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:43:33.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:43:33.086 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:43:33.086 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T18:43:33.155 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:43:33.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:43:33.261 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:43:33.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:43:33.386 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:43:33.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T18:43:33.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-02-19T18:43:33.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-02-19T18:43:33.501 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T18:43:33.552 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T18:43:33.879 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T18:43:33.903 INFO:teuthology.task.internal:Starting timer... 2024-02-19T18:43:33.903 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T18:43:33.970 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T18:43:33.990 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T18:43:33.991 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-02-19T18:43:34.079 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-02-19T18:43:34.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-02-19T18:43:34.164 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-02-19T18:43:34.166 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T18:43:34.166 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-02-19T18:43:34.186 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-02-19T18:43:34.186 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-02-19T18:43:34.213 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-02-19T18:43:34.213 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-02-19T18:43:34.213 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T18:43:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:43:34.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T18:43:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:43:34.287 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T18:43:34.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-02-19T18:43:34.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-02-19T18:43:34.415 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T18:43:34.426 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T18:43:34.453 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T18:43:34.458 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T18:49:04.144 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-02-19T18:49:04.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:49:04.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:49:04.234 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-02-19T18:49:04.314 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-02-19T18:49:04.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T18:49:04.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:49:04.399 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-02-19T18:49:04.487 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T18:49:04.487 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T18:49:04.497 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T18:49:04.497 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T18:49:04.498 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T18:49:04.500 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T18:49:04.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T18:49:04.529 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T18:49:04.545 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T18:49:04.558 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T18:49:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-02-19T18:49:04.575 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T18:49:04.588 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-02-19T18:49:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T18:49:04.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-02-19T18:49:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-02-19T18:49:04.621 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T18:49:04.635 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T18:49:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T18:49:04.670 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-02-19T18:49:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T18:49:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-02-19T18:49:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.704 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-02-19T18:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T18:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-02-19T18:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:49:04.709 INFO:teuthology.run_tasks:Running task install... 2024-02-19T18:49:04.717 DEBUG:teuthology.task.install:project ceph 2024-02-19T18:49:04.717 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'}} 2024-02-19T18:49:04.717 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-19T18:49:04.717 INFO:teuthology.task.install:Using flavor: default 2024-02-19T18:49:04.722 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T18:49:04.722 INFO:teuthology.task.install:extra packages: [] 2024-02-19T18:49:04.722 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-02-19T18:49:04.723 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:49:04.723 INFO:teuthology.packaging:ref: None 2024-02-19T18:49:04.723 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:49:04.723 INFO:teuthology.packaging:branch: None 2024-02-19T18:49:04.723 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:49:05.040 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.1^{} -> None 2024-02-19T18:49:05.041 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:49:06.690 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:49:06.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:49:06.694 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-02-19T18:49:06.694 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:49:06.694 INFO:teuthology.packaging:ref: None 2024-02-19T18:49:06.694 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:49:06.694 INFO:teuthology.packaging:branch: None 2024-02-19T18:49:06.694 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:49:06.695 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:49:06.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:49:06.841 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-02-19T18:49:06.841 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-02-19T18:49:06.847 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-02-19T18:49:06.847 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-02-19T18:49:06.880 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T18:49:06.880 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:49:06.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T18:49:06.883 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T18:49:06.883 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:49:06.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T18:49:06.916 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T18:49:06.916 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:49:06.916 INFO:teuthology.packaging:ref: None 2024-02-19T18:49:06.916 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:49:06.916 INFO:teuthology.packaging:branch: None 2024-02-19T18:49:06.916 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:49:06.916 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T18:49:06.917 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T18:49:06.917 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:49:06.918 INFO:teuthology.packaging:ref: None 2024-02-19T18:49:06.918 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:49:06.918 INFO:teuthology.packaging:branch: None 2024-02-19T18:49:06.918 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:49:06.918 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T18:49:06.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T18:49:06.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T18:49:07.072 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T18:49:07.073 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T18:49:07.141 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-02-19T18:49:07.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-02-19T18:49:07.143 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-02-19T18:49:07.144 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-02-19T18:49:07.394 INFO:teuthology.orchestra.run.smithi086.stdout:74 files removed 2024-02-19T18:49:07.402 INFO:teuthology.orchestra.run.smithi039.stdout:74 files removed 2024-02-19T18:49:07.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-02-19T18:49:07.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2024-02-19T18:49:07.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 548 kB/s | 76 kB 00:00 2024-02-19T18:49:07.906 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 507 kB/s | 76 kB 00:00 2024-02-19T18:49:08.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 70 kB/s | 9.4 kB 00:00 2024-02-19T18:49:08.098 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 66 kB/s | 9.4 kB 00:00 2024-02-19T18:49:08.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 15 kB/s | 2.2 kB 00:00 2024-02-19T18:49:08.273 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 16 kB/s | 2.2 kB 00:00 2024-02-19T18:49:09.235 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-02-19T18:49:09.246 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.7 MB 00:00 2024-02-19T18:49:10.910 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-19T18:49:10.941 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-02-19T18:49:11.689 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-02-19T18:49:15.392 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-02-19T18:49:22.734 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-02-19T18:49:27.591 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-02-19T18:49:29.316 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-02-19T18:49:41.561 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 20 MB 00:12 2024-02-19T18:49:46.350 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-02-19T18:49:48.902 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-02-19T18:49:48.982 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T18:49:48.983 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-02-19T18:49:48.984 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:Install 28 Packages 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 53 M 2024-02-19T18:49:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:49:49.210 INFO:teuthology.orchestra.run.smithi086.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 29 MB/s | 5.2 MB 00:00 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi086.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 1.4 MB/s | 24 kB 00:00 2024-02-19T18:49:49.260 INFO:teuthology.orchestra.run.smithi086.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-02-19T18:49:49.294 INFO:teuthology.orchestra.run.smithi086.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-02-19T18:49:49.411 INFO:teuthology.orchestra.run.smithi086.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 20 MB/s | 7.7 MB 00:00 2024-02-19T18:49:49.427 INFO:teuthology.orchestra.run.smithi086.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.7 MB/s | 45 kB 00:00 2024-02-19T18:49:49.453 INFO:teuthology.orchestra.run.smithi086.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-02-19T18:49:49.478 INFO:teuthology.orchestra.run.smithi086.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-02-19T18:49:49.503 INFO:teuthology.orchestra.run.smithi086.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-02-19T18:49:49.570 INFO:teuthology.orchestra.run.smithi086.stdout:(10/30): librgw2-18.2.1-0.el9.x86_64.rpm 16 MB/s | 4.5 MB 00:00 2024-02-19T18:49:49.596 INFO:teuthology.orchestra.run.smithi086.stdout:(11/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 3.2 MB/s | 297 kB 00:00 2024-02-19T18:49:49.621 INFO:teuthology.orchestra.run.smithi086.stdout:(12/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-02-19T18:49:49.871 INFO:teuthology.orchestra.run.smithi086.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-02-19T18:49:49.997 INFO:teuthology.orchestra.run.smithi086.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 282 kB/s | 106 kB 00:00 2024-02-19T18:49:50.022 INFO:teuthology.orchestra.run.smithi086.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 397 kB/s | 159 kB 00:00 2024-02-19T18:49:50.047 INFO:teuthology.orchestra.run.smithi086.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 255 kB/s | 44 kB 00:00 2024-02-19T18:49:50.147 INFO:teuthology.orchestra.run.smithi086.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-02-19T18:49:50.173 INFO:teuthology.orchestra.run.smithi086.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-02-19T18:49:50.198 INFO:teuthology.orchestra.run.smithi086.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 274 kB/s | 41 kB 00:00 2024-02-19T18:49:50.864 INFO:teuthology.orchestra.run.smithi086.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 154 kB/s | 111 kB 00:00 2024-02-19T18:49:51.006 INFO:teuthology.orchestra.run.smithi086.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-02-19T18:49:51.273 INFO:teuthology.orchestra.run.smithi086.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 182 kB/s | 49 kB 00:00 2024-02-19T18:49:51.615 INFO:teuthology.orchestra.run.smithi086.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-02-19T18:49:51.649 INFO:teuthology.orchestra.run.smithi086.stdout:(24/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 213 kB/s | 309 kB 00:01 2024-02-19T18:49:52.349 INFO:teuthology.orchestra.run.smithi086.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 274 kB/s | 192 kB 00:00 2024-02-19T18:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 317 kB/s | 838 kB 00:02 2024-02-19T18:49:54.324 INFO:teuthology.orchestra.run.smithi086.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 49 MB/s | 3.3 MB 00:00 2024-02-19T18:49:54.383 INFO:teuthology.orchestra.run.smithi086.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 52 MB/s | 3.0 MB 00:00 2024-02-19T18:49:56.458 INFO:teuthology.orchestra.run.smithi086.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 394 kB/s | 1.6 MB 00:04 2024-02-19T18:50:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 294 kB/s | 4.4 MB 00:15 2024-02-19T18:50:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.2 MB/s | 53 MB 00:16 2024-02-19T18:50:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:50:05.642 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:50:05.642 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:50:06.088 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:50:06.088 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:50:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:50:07.541 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-02-19T18:50:07.579 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-02-19T18:50:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-02-19T18:50:07.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-02-19T18:50:07.986 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:50:08.078 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:50:08.158 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:50:08.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:50:08.275 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:50:08.312 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-02-19T18:50:08.355 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:50:08.422 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:50:08.448 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:50:08.497 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:50:08.612 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-02-19T18:50:08.653 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-02-19T18:50:08.723 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-02-19T18:50:08.771 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-02-19T18:50:08.797 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-02-19T18:50:08.833 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-02-19T18:50:08.866 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-02-19T18:50:09.164 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-02-19T18:50:09.188 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:50:09.277 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:50:09.309 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-02-19T18:50:09.354 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-02-19T18:50:09.753 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-02-19T18:50:09.782 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:50:09.861 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:50:09.928 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-02-19T18:50:09.959 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-02-19T18:50:12.037 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:50:12.072 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:50:12.612 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:50:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:50:12.768 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:50:12.769 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T18:50:12.769 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T18:50:12.769 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:12.802 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:50:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-02-19T18:50:21.841 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:22.605 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:50:22.630 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:22.631 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:50:22.715 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:50:22.716 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:50:23.813 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-02-19T18:50:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-02-19T18:50:23.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-02-19T18:50:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: fmt-8.1.1-5.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T18:50:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.925 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:24.926 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:25.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-02-19T18:50:25.866 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:39 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:26.028 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:50:26.029 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-02-19T18:50:26.030 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:26.031 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 41 M 2024-02-19T18:50:26.031 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 167 M 2024-02-19T18:50:26.031 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:50:26.163 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 165 kB 00:00 2024-02-19T18:50:26.213 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-02-19T18:50:26.238 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 836 kB/s | 63 kB 00:00 2024-02-19T18:50:26.272 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-02-19T18:50:26.297 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-02-19T18:50:26.789 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-18.2.1-0.el9.x86_64.rpm 52 MB/s | 40 MB 00:00 2024-02-19T18:50:26.793 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:26.793 INFO:teuthology.orchestra.run.smithi086.stdout:Total 53 MB/s | 41 MB 00:00 2024-02-19T18:50:26.793 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:50:26.812 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:50:26.812 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:50:27.158 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:50:27.158 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:50:27.706 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:50:27.771 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-19T18:50:27.805 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-19T18:50:27.858 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-19T18:50:27.907 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-19T18:50:27.934 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:50:31.680 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:50:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:50:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:50:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-02-19T18:50:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-02-19T18:50:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-19T18:50:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-19T18:50:34.023 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:34.409 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:34.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-02-19T18:50:35.283 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:49 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:35.465 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:35.467 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:35.467 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:50:35.467 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:35.467 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:50:35.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T18:50:35.468 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T18:50:35.469 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout:Install 33 Packages 2024-02-19T18:50:35.470 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:35.471 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 30 M 2024-02-19T18:50:35.471 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 110 M 2024-02-19T18:50:35.471 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:50:35.625 INFO:teuthology.orchestra.run.smithi086.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 60 kB/s | 6.4 kB 00:00 2024-02-19T18:50:35.675 INFO:teuthology.orchestra.run.smithi086.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.2 MB/s | 1.4 MB 00:00 2024-02-19T18:50:35.726 INFO:teuthology.orchestra.run.smithi086.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 10 MB/s | 2.1 MB 00:00 2024-02-19T18:50:35.751 INFO:teuthology.orchestra.run.smithi086.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-02-19T18:50:35.818 INFO:teuthology.orchestra.run.smithi086.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-02-19T18:50:36.085 INFO:teuthology.orchestra.run.smithi086.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-02-19T18:50:36.160 INFO:teuthology.orchestra.run.smithi086.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 99 kB/s | 32 kB 00:00 2024-02-19T18:50:36.202 INFO:teuthology.orchestra.run.smithi086.stdout:(8/33): python3-mako-1.1.4-6.el9.noarch.rpm 320 kB/s | 144 kB 00:00 2024-02-19T18:50:36.253 INFO:teuthology.orchestra.run.smithi086.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 285 kB/s | 47 kB 00:00 2024-02-19T18:50:36.294 INFO:teuthology.orchestra.run.smithi086.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 288 kB/s | 39 kB 00:00 2024-02-19T18:50:36.553 INFO:teuthology.orchestra.run.smithi086.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 686 kB/s | 241 kB 00:00 2024-02-19T18:50:37.487 INFO:teuthology.orchestra.run.smithi086.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 86 kB/s | 103 kB 00:01 2024-02-19T18:50:37.587 INFO:teuthology.orchestra.run.smithi086.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 120 kB/s | 124 kB 00:01 2024-02-19T18:50:37.679 INFO:teuthology.orchestra.run.smithi086.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 821 kB/s | 1.1 MB 00:01 2024-02-19T18:50:37.729 INFO:teuthology.orchestra.run.smithi086.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 467 kB/s | 113 kB 00:00 2024-02-19T18:50:37.846 INFO:teuthology.orchestra.run.smithi086.stdout:(16/33): python3-urllib3-1.26.5-5.el9.noarch.rp 723 kB/s | 187 kB 00:00 2024-02-19T18:50:38.088 INFO:teuthology.orchestra.run.smithi086.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 106 kB/s | 43 kB 00:00 2024-02-19T18:50:38.280 INFO:teuthology.orchestra.run.smithi086.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 312 kB/s | 172 kB 00:00 2024-02-19T18:50:38.305 INFO:teuthology.orchestra.run.smithi086.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 49 kB/s | 11 kB 00:00 2024-02-19T18:50:38.456 INFO:teuthology.orchestra.run.smithi086.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 590 kB/s | 358 kB 00:00 2024-02-19T18:50:38.472 INFO:teuthology.orchestra.run.smithi086.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 92 kB/s | 18 kB 00:00 2024-02-19T18:50:38.489 INFO:teuthology.orchestra.run.smithi086.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 126 kB/s | 23 kB 00:00 2024-02-19T18:50:38.573 INFO:teuthology.orchestra.run.smithi086.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-02-19T18:50:38.606 INFO:teuthology.orchestra.run.smithi086.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-02-19T18:50:38.707 INFO:teuthology.orchestra.run.smithi086.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 213 kB/s | 46 kB 00:00 2024-02-19T18:50:38.732 INFO:teuthology.orchestra.run.smithi086.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 497 kB/s | 79 kB 00:00 2024-02-19T18:50:38.932 INFO:teuthology.orchestra.run.smithi086.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 837 kB/s | 272 kB 00:00 2024-02-19T18:50:38.949 INFO:teuthology.orchestra.run.smithi086.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 76 kB/s | 16 kB 00:00 2024-02-19T18:50:39.041 INFO:teuthology.orchestra.run.smithi086.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 291 kB/s | 90 kB 00:00 2024-02-19T18:50:39.058 INFO:teuthology.orchestra.run.smithi086.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-02-19T18:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 655 kB/s | 230 kB 00:00 2024-02-19T18:50:39.408 INFO:teuthology.orchestra.run.smithi086.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 57 kB/s | 20 kB 00:00 2024-02-19T18:50:40.450 INFO:teuthology.orchestra.run.smithi086.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 303 kB/s | 427 kB 00:01 2024-02-19T18:50:40.456 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:40.457 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.0 MB/s | 30 MB 00:04 2024-02-19T18:50:40.457 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:50:40.497 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:50:40.497 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:50:40.884 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:50:40.885 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:50:42.110 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:50:42.169 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-19T18:50:42.201 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-19T18:50:42.234 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-19T18:50:42.325 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-19T18:50:42.406 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-19T18:50:42.447 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-19T18:50:42.518 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-19T18:50:42.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-19T18:50:42.611 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-19T18:50:42.660 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-19T18:50:42.718 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-19T18:50:42.899 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-19T18:50:42.949 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-19T18:50:42.993 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-19T18:50:43.030 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-19T18:50:43.060 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-19T18:50:43.136 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-19T18:50:43.199 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:50:43.204 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-19T18:50:43.244 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-19T18:50:43.279 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-19T18:50:43.314 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-19T18:50:43.405 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-19T18:50:43.445 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-19T18:50:43.515 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-19T18:50:43.653 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-19T18:50:43.769 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:50:43.923 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-02-19T18:50:43.965 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:50:43.992 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:50:43.992 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:43.992 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T18:50:43.992 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:50:43.992 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:50:43.993 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:45.491 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:50:45.517 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:50:45.517 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:45.518 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T18:50:45.518 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:50:45.518 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:50:45.518 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:50:45.990 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:50:45.990 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:45.990 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T18:50:45.990 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:50:45.990 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:50:45.990 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:46.193 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:50:46.215 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:50:46.215 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:46.215 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T18:50:46.216 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:50:46.216 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:50:46.216 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:46.282 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:50:46.375 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 218 kB/s | 19 MB 01:29 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-19T18:50:48.036 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-19T18:50:48.037 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-19T18:50:48.038 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-02-19T18:50:48.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T18:50:48.917 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:48.918 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:49.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-02-19T18:50:49.816 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:03 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:49.930 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:49.980 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:49.981 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:50:49.981 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:50.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-02-19T18:50:50.484 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-02-19T18:50:50.622 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:04 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:50.788 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:50.788 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:50.788 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:50.788 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:50:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 221 k 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 213 k 2024-02-19T18:50:50.789 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:50:50.866 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.8 MB/s | 221 kB 00:00 2024-02-19T18:50:50.866 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:50.867 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.8 MB/s | 221 kB 00:00 2024-02-19T18:50:50.867 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:50:50.871 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:50:50.872 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:50:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:50:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:50:50.942 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:50:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:50:51.438 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:50:51.825 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:50:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:50:52.119 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:52.119 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:50:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:50:52.120 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:52.120 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:52.143 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2024-02-19T18:50:52.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T18:50:52.891 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:06 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:53.057 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:53.058 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:53.058 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-02-19T18:50:53.058 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:53.058 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:50:53.058 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-02-19T18:50:53.058 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 142 k 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 395 k 2024-02-19T18:50:53.059 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:50:53.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 2.1 MB/s | 142 kB 00:00 2024-02-19T18:50:53.126 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:53.127 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.0 MB/s | 142 kB 00:00 2024-02-19T18:50:53.127 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:50:53.134 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:50:53.134 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:50:53.186 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:50:53.186 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:50:53.324 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:50:53.368 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:53.392 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:53.392 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:53.392 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T18:50:53.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:54.368 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:54.368 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:54.368 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:50:54.368 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:54.368 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:54.368 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:54.582 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 20 MB 00:02 2024-02-19T18:50:54.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-02-19T18:50:55.201 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:55.314 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:55.366 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:55.367 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:50:55.367 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:50:55.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T18:50:56.010 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:50:56.176 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T18:50:56.177 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout:Install 7 Packages 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.1 M 2024-02-19T18:50:56.178 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 8.5 M 2024-02-19T18:50:56.179 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:50:56.628 INFO:teuthology.orchestra.run.smithi086.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 222 kB/s | 23 kB 00:00 2024-02-19T18:50:56.645 INFO:teuthology.orchestra.run.smithi086.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 121 kB/s | 15 kB 00:00 2024-02-19T18:50:56.696 INFO:teuthology.orchestra.run.smithi086.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 10 MB/s | 1.8 MB 00:00 2024-02-19T18:50:56.929 INFO:teuthology.orchestra.run.smithi086.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 30 kB/s | 9.0 kB 00:00 2024-02-19T18:50:57.104 INFO:teuthology.orchestra.run.smithi086.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 75 kB/s | 31 kB 00:00 2024-02-19T18:50:57.188 INFO:teuthology.orchestra.run.smithi086.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 75 kB/s | 41 kB 00:00 2024-02-19T18:50:58.980 INFO:teuthology.orchestra.run.smithi086.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 92 kB/s | 188 kB 00:02 2024-02-19T18:50:58.982 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:58.982 INFO:teuthology.orchestra.run.smithi086.stdout:Total 751 kB/s | 2.1 MB 00:02 2024-02-19T18:50:58.983 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:50:58.991 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:50:58.991 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:50:59.066 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:50:59.067 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:50:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:50:59.439 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-19T18:50:59.473 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-19T18:50:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-19T18:50:59.568 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-19T18:50:59.595 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-19T18:50:59.597 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-02-19T18:50:59.833 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-02-19T18:50:59.867 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-19T18:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:00.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T18:51:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:01.680 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:51:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T18:51:01.681 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-02-19T18:51:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:02.252 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:51:02.261 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T18:51:02.262 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout:Install 28 Packages 2024-02-19T18:51:02.263 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2024-02-19T18:51:02.264 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:02.264 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 53 M 2024-02-19T18:51:02.264 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:51:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 39 MB/s | 7.4 MB 00:00 2024-02-19T18:51:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 100 kB/s | 32 kB 00:00 2024-02-19T18:51:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 115 kB/s | 17 kB 00:00 2024-02-19T18:51:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 409 kB/s | 34 kB 00:00 2024-02-19T18:51:02.971 INFO:teuthology.orchestra.run.smithi039.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 24 MB/s | 5.2 MB 00:00 2024-02-19T18:51:02.996 INFO:teuthology.orchestra.run.smithi039.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 973 kB/s | 24 kB 00:00 2024-02-19T18:51:03.030 INFO:teuthology.orchestra.run.smithi039.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-02-19T18:51:03.063 INFO:teuthology.orchestra.run.smithi039.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-02-19T18:51:03.181 INFO:teuthology.orchestra.run.smithi039.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 18 MB/s | 7.7 MB 00:00 2024-02-19T18:51:03.199 INFO:teuthology.orchestra.run.smithi039.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.7 MB/s | 45 kB 00:00 2024-02-19T18:51:03.222 INFO:teuthology.orchestra.run.smithi039.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-02-19T18:51:03.248 INFO:teuthology.orchestra.run.smithi039.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-02-19T18:51:03.307 INFO:teuthology.orchestra.run.smithi039.stdout:(9/30): librgw2-18.2.1-0.el9.x86_64.rpm 18 MB/s | 4.5 MB 00:00 2024-02-19T18:51:03.332 INFO:teuthology.orchestra.run.smithi039.stdout:(10/30): python3-rados-18.2.1-0.el9.x86_64.rpm 3.7 MB/s | 321 kB 00:00 2024-02-19T18:51:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.0 MB/s | 4.7 MB 00:00 2024-02-19T18:51:03.357 INFO:teuthology.orchestra.run.smithi039.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-02-19T18:51:03.382 INFO:teuthology.orchestra.run.smithi039.stdout:(12/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 3.8 MB/s | 297 kB 00:00 2024-02-19T18:51:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.1 MB/s | 4.8 MB 00:00 2024-02-19T18:51:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-02-19T18:51:03.558 INFO:teuthology.orchestra.run.smithi039.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 23 MB/s | 18 MB 00:00 2024-02-19T18:51:03.858 INFO:teuthology.orchestra.run.smithi039.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 211 kB/s | 106 kB 00:00 2024-02-19T18:51:03.883 INFO:teuthology.orchestra.run.smithi039.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 160 kB/s | 44 kB 00:00 2024-02-19T18:51:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 802 kB 00:00 2024-02-19T18:51:03.909 INFO:teuthology.orchestra.run.smithi039.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 302 kB/s | 159 kB 00:00 2024-02-19T18:51:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.7 MB/s | 3.0 MB 00:01 2024-02-19T18:51:04.134 INFO:teuthology.orchestra.run.smithi039.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 661 kB 00:00 2024-02-19T18:51:04.167 INFO:teuthology.orchestra.run.smithi039.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 994 kB/s | 282 kB 00:00 2024-02-19T18:51:04.193 INFO:teuthology.orchestra.run.smithi039.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 145 kB/s | 41 kB 00:00 2024-02-19T18:51:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-02-19T18:51:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 467 kB/s | 194 kB 00:00 2024-02-19T18:51:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 803 kB/s | 245 kB 00:00 2024-02-19T18:51:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:Total 14 MB/s | 37 MB 00:02 2024-02-19T18:51:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:04.502 INFO:teuthology.orchestra.run.smithi039.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 302 kB/s | 111 kB 00:00 2024-02-19T18:51:04.593 INFO:teuthology.orchestra.run.smithi039.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-02-19T18:51:04.669 INFO:teuthology.orchestra.run.smithi039.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 616 kB/s | 309 kB 00:00 2024-02-19T18:51:04.711 INFO:teuthology.orchestra.run.smithi039.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-02-19T18:51:04.803 INFO:teuthology.orchestra.run.smithi039.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-02-19T18:51:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:04.920 INFO:teuthology.orchestra.run.smithi039.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-02-19T18:51:05.287 INFO:teuthology.orchestra.run.smithi039.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-02-19T18:51:05.321 INFO:teuthology.orchestra.run.smithi039.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-02-19T18:51:05.362 INFO:teuthology.orchestra.run.smithi039.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-02-19T18:51:05.446 INFO:teuthology.orchestra.run.smithi039.stdout:(29/30): librados2-18.2.1-0.el9.x86_64.rpm 21 MB/s | 3.3 MB 00:00 2024-02-19T18:51:05.496 INFO:teuthology.orchestra.run.smithi039.stdout:(30/30): librbd1-18.2.1-0.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-02-19T18:51:05.501 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:05.501 INFO:teuthology.orchestra.run.smithi039.stdout:Total 16 MB/s | 53 MB 00:03 2024-02-19T18:51:05.501 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:51:05.541 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:51:05.542 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:51:05.560 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:05.645 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-19T18:51:05.676 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-19T18:51:05.770 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:51:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-19T18:51:05.990 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:51:06.016 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:51:06.016 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:51:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:51:06.109 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-19T18:51:06.757 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-19T18:51:06.865 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-19T18:51:07.561 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:51:07.639 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-02-19T18:51:07.668 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-02-19T18:51:07.720 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-02-19T18:51:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-02-19T18:51:08.103 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-19T18:51:08.183 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:51:08.276 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:51:08.359 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:51:08.510 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:51:08.573 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:51:08.602 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-02-19T18:51:08.646 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:51:08.725 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:51:08.754 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:51:08.820 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:51:08.931 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-02-19T18:51:08.972 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-02-19T18:51:09.011 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-02-19T18:51:09.044 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-02-19T18:51:09.071 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-02-19T18:51:09.106 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-02-19T18:51:09.119 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-19T18:51:09.139 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-02-19T18:51:09.143 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:51:09.454 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-02-19T18:51:09.486 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:51:09.578 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:51:09.607 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-02-19T18:51:09.654 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-02-19T18:51:10.074 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-02-19T18:51:10.097 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:51:10.160 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:51:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-02-19T18:51:10.258 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-02-19T18:51:11.715 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-19T18:51:11.716 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-19T18:51:12.119 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-19T18:51:12.119 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:12.119 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:12.119 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-02-19T18:51:12.119 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:12.120 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:12.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-02-19T18:51:12.530 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:51:12.563 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:51:13.069 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:13.131 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:51:13.167 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:51:13.236 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:13.237 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:13.237 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T18:51:13.238 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-02-19T18:51:13.239 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:13.240 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2024-02-19T18:51:13.240 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-02-19T18:51:13.240 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:13.256 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:51:13.256 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T18:51:13.256 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T18:51:13.256 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:13.293 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:51:15.301 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 726 kB/s | 50 kB 00:00 2024-02-19T18:51:15.477 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 100 kB/s | 24 kB 00:00 2024-02-19T18:51:15.552 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 57 kB/s | 18 kB 00:00 2024-02-19T18:51:15.660 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 721 kB/s | 132 kB 00:00 2024-02-19T18:51:15.719 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 211 kB 00:00 2024-02-19T18:51:15.744 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 445 kB/s | 174 kB 00:00 2024-02-19T18:51:15.761 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 43 kB 00:00 2024-02-19T18:51:15.970 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2024-02-19T18:51:15.986 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-02-19T18:51:16.203 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2024-02-19T18:51:16.229 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 434 kB/s | 203 kB 00:00 2024-02-19T18:51:16.412 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-02-19T18:51:16.705 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-02-19T18:51:16.708 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:16.708 INFO:teuthology.orchestra.run.smithi086.stdout:Total 604 kB/s | 2.0 MB 00:03 2024-02-19T18:51:16.708 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:16.723 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:16.723 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:16.829 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:16.830 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:17.518 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-19T18:51:17.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-19T18:51:17.627 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-19T18:51:17.664 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-19T18:51:17.714 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-19T18:51:17.807 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-19T18:51:17.908 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-19T18:51:17.956 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-19T18:51:18.676 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-19T18:51:18.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-19T18:51:18.754 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-19T18:51:18.801 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-19T18:51:18.824 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-19T18:51:19.625 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-19T18:51:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-19T18:51:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-19T18:51:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-19T18:51:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-19T18:51:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-19T18:51:19.626 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-19T18:51:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-19T18:51:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:19.992 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-02-19T18:51:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T18:51:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:19.993 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:20.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T18:51:20.848 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:21.015 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:21.015 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:21.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:21.017 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-02-19T18:51:21.017 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:21.017 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.8 M 2024-02-19T18:51:21.017 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-02-19T18:51:21.017 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:21.446 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 936 kB/s | 132 kB 00:00 2024-02-19T18:51:21.705 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 569 kB/s | 227 kB 00:00 2024-02-19T18:51:22.147 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.9 MB/s | 5.8 MB 00:00 2024-02-19T18:51:22.222 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 112 kB/s | 58 kB 00:00 2024-02-19T18:51:23.348 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-02-19T18:51:23.349 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:23.622 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 35 kB/s | 51 kB 00:01 2024-02-19T18:51:24.118 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:51:24.141 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:51:24.142 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:24.142 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T18:51:24.142 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:51:24.142 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:51:24.142 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:24.144 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:51:24.228 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:51:24.229 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:51:25.260 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-02-19T18:51:25.261 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-02-19T18:51:25.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:51:25.263 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-02-19T18:51:25.263 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.450 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: fmt-8.1.1-5.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.451 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-3.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:26.452 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:26.664 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 105 kB/s | 548 kB 00:05 2024-02-19T18:51:26.666 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:26.666 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.2 MB/s | 6.8 MB 00:05 2024-02-19T18:51:26.667 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:26.676 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:26.676 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:26.770 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:26.771 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:26.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2024-02-19T18:51:27.067 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:27.243 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-19T18:51:27.289 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-19T18:51:27.378 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:28 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:27.555 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:27.555 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:27.555 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:27.556 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2024-02-19T18:51:27.557 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:27.557 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 41 M 2024-02-19T18:51:27.557 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 167 M 2024-02-19T18:51:27.557 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:51:27.731 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 958 kB/s | 165 kB 00:00 2024-02-19T18:51:27.797 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-19T18:51:27.883 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-19T18:51:27.944 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-19T18:51:27.968 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:51:28.099 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 554 kB/s | 300 kB 00:00 2024-02-19T18:51:28.125 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 159 kB/s | 63 kB 00:00 2024-02-19T18:51:28.192 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2024-02-19T18:51:28.225 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 217 kB 00:00 2024-02-19T18:51:28.634 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): ceph-test-18.2.1-0.el9.x86_64.rpm 37 MB/s | 40 MB 00:01 2024-02-19T18:51:28.637 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:28.637 INFO:teuthology.orchestra.run.smithi039.stdout:Total 38 MB/s | 41 MB 00:01 2024-02-19T18:51:28.637 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:51:28.659 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:51:28.659 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:51:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:51:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-02-19T18:51:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-19T18:51:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-19T18:51:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-19T18:51:28.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-19T18:51:29.027 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:51:29.027 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:51:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-19T18:51:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:29.116 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:51:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T18:51:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T18:51:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T18:51:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T18:51:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T18:51:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:29.118 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:29.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-02-19T18:51:29.575 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:51:29.632 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-19T18:51:29.666 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-19T18:51:29.718 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-19T18:51:29.768 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-19T18:51:29.795 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:51:29.954 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:30.119 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-02-19T18:51:30.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 918 k 2024-02-19T18:51:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.7 M 2024-02-19T18:51:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:30.265 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 5.7 MB/s | 839 kB 00:00 2024-02-19T18:51:30.540 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 188 kB/s | 79 kB 00:00 2024-02-19T18:51:30.541 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:30.541 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.1 MB/s | 918 kB 00:00 2024-02-19T18:51:30.542 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:30.562 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:30.562 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:30.638 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:30.638 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:30.821 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:30.962 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-19T18:51:31.030 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:51:31.712 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:51:31.712 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-02-19T18:51:32.022 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-19T18:51:32.022 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:32.022 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:32.022 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-19T18:51:32.022 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:32.022 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:32.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-02-19T18:51:32.889 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:46 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:33.054 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:33.054 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 127 k 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2024-02-19T18:51:33.055 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:33.177 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.0 MB/s | 127 kB 00:00 2024-02-19T18:51:33.177 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:33.177 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.0 MB/s | 127 kB 00:00 2024-02-19T18:51:33.177 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:33.182 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:33.183 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:33.237 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:33.237 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:33.474 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:51:33.807 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:51:33.930 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:34.248 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:34.249 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:34.249 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:34.249 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:34.249 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:34.249 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:34.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-02-19T18:51:35.127 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:49 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:35.246 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:35.300 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:35.300 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:35.300 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:35.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-02-19T18:51:35.929 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:51:35.929 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-02-19T18:51:35.929 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-02-19T18:51:35.929 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-19T18:51:35.929 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-19T18:51:35.930 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-19T18:51:35.940 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:49 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:36.107 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:36.108 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:36.108 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:36.108 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-02-19T18:51:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 k 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 139 k 2024-02-19T18:51:36.109 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:36.188 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 392 kB/s | 31 kB 00:00 2024-02-19T18:51:36.191 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:36.192 INFO:teuthology.orchestra.run.smithi086.stdout:Total 386 kB/s | 31 kB 00:00 2024-02-19T18:51:36.192 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:36.192 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:36.192 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:36.226 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:36.226 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:36.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-19T18:51:36.371 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:36.371 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:51:36.371 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-02-19T18:51:36.371 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T18:51:36.371 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T18:51:36.372 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:36.372 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:36.639 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2024-02-19T18:51:36.834 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:37.105 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:37.106 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:37.106 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:37.106 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:37.106 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:37.106 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:37.223 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:38 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:37.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-02-19T18:51:37.407 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:37.409 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:37.409 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T18:51:37.410 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T18:51:37.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:37.412 INFO:teuthology.orchestra.run.smithi039.stdout:Install 33 Packages 2024-02-19T18:51:37.413 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:37.413 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 30 M 2024-02-19T18:51:37.413 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 110 M 2024-02-19T18:51:37.413 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:51:37.852 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:51 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:37.953 INFO:teuthology.orchestra.run.smithi039.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 55 kB/s | 6.4 kB 00:00 2024-02-19T18:51:37.964 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:38.015 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:38.017 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:38.017 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:38.028 INFO:teuthology.orchestra.run.smithi039.stdout:(2/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-02-19T18:51:38.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-02-19T18:51:38.085 INFO:teuthology.orchestra.run.smithi039.stdout:(3/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 5.8 MB/s | 1.4 MB 00:00 2024-02-19T18:51:38.137 INFO:teuthology.orchestra.run.smithi039.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 4.6 MB/s | 242 kB 00:00 2024-02-19T18:51:38.203 INFO:teuthology.orchestra.run.smithi039.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-02-19T18:51:38.504 INFO:teuthology.orchestra.run.smithi039.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 37 MB/s | 18 MB 00:00 2024-02-19T18:51:38.658 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:52 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:38.696 INFO:teuthology.orchestra.run.smithi039.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 68 kB/s | 32 kB 00:00 2024-02-19T18:51:38.771 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:38.779 INFO:teuthology.orchestra.run.smithi039.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 173 kB/s | 47 kB 00:00 2024-02-19T18:51:38.824 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:38.825 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:38.825 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:38.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-02-19T18:51:38.905 INFO:teuthology.orchestra.run.smithi039.stdout:(9/33): python3-mako-1.1.4-6.el9.noarch.rpm 188 kB/s | 144 kB 00:00 2024-02-19T18:51:38.921 INFO:teuthology.orchestra.run.smithi039.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 171 kB/s | 39 kB 00:00 2024-02-19T18:51:39.064 INFO:teuthology.orchestra.run.smithi039.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 848 kB/s | 241 kB 00:00 2024-02-19T18:51:39.214 INFO:teuthology.orchestra.run.smithi039.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 3.7 MB/s | 1.1 MB 00:00 2024-02-19T18:51:39.239 INFO:teuthology.orchestra.run.smithi039.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 324 kB/s | 103 kB 00:00 2024-02-19T18:51:39.264 INFO:teuthology.orchestra.run.smithi039.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 619 kB/s | 124 kB 00:00 2024-02-19T18:51:39.464 INFO:teuthology.orchestra.run.smithi039.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 452 kB/s | 113 kB 00:00 2024-02-19T18:51:39.487 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:53 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:39.588 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:39.643 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:39.644 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:39.644 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:39.690 INFO:teuthology.orchestra.run.smithi039.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 102 kB/s | 43 kB 00:00 2024-02-19T18:51:39.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-02-19T18:51:39.998 INFO:teuthology.orchestra.run.smithi039.stdout:(17/33): python3-urllib3-1.26.5-5.el9.noarch.rp 255 kB/s | 187 kB 00:00 2024-02-19T18:51:40.074 INFO:teuthology.orchestra.run.smithi039.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 283 kB/s | 172 kB 00:00 2024-02-19T18:51:40.099 INFO:teuthology.orchestra.run.smithi039.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 876 kB/s | 358 kB 00:00 2024-02-19T18:51:40.183 INFO:teuthology.orchestra.run.smithi039.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-02-19T18:51:40.233 INFO:teuthology.orchestra.run.smithi039.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 218 kB/s | 23 kB 00:00 2024-02-19T18:51:40.283 INFO:teuthology.orchestra.run.smithi039.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-02-19T18:51:40.308 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:54 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:40.317 INFO:teuthology.orchestra.run.smithi039.stdout:(23/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 34 kB/s | 11 kB 00:00 2024-02-19T18:51:40.342 INFO:teuthology.orchestra.run.smithi039.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-02-19T18:51:40.404 INFO:teuthology.orchestra.run.smithi039.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 396 kB/s | 46 kB 00:00 2024-02-19T18:51:40.427 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:40.481 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:40.482 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:40.482 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:40.484 INFO:teuthology.orchestra.run.smithi039.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-02-19T18:51:40.510 INFO:teuthology.orchestra.run.smithi039.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-02-19T18:51:40.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-02-19T18:51:40.585 INFO:teuthology.orchestra.run.smithi039.stdout:(28/33): python3-more-itertools-8.12.0-2.el9.no 295 kB/s | 79 kB 00:00 2024-02-19T18:51:40.610 INFO:teuthology.orchestra.run.smithi039.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-02-19T18:51:40.635 INFO:teuthology.orchestra.run.smithi039.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-02-19T18:51:40.744 INFO:teuthology.orchestra.run.smithi039.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-02-19T18:51:40.786 INFO:teuthology.orchestra.run.smithi039.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-02-19T18:51:40.852 INFO:teuthology.orchestra.run.smithi039.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 859 kB/s | 230 kB 00:00 2024-02-19T18:51:40.860 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:40.860 INFO:teuthology.orchestra.run.smithi039.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-02-19T18:51:40.861 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:51:40.903 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:51:40.903 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:51:41.124 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:55 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:41.238 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:41.287 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:51:41.288 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:51:41.292 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:41.293 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:41.293 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:41.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-02-19T18:51:41.926 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:55 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:42.039 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:42.091 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:42.092 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-02-19T18:51:42.092 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:42.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-02-19T18:51:42.531 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:51:42.588 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-19T18:51:42.621 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-19T18:51:42.662 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-19T18:51:42.728 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:56 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-19T18:51:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-19T18:51:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-19T18:51:42.893 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:42.894 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 86 k 2024-02-19T18:51:42.895 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 226 k 2024-02-19T18:51:42.895 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:42.965 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-19T18:51:42.966 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-02-19T18:51:42.967 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:42.967 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-02-19T18:51:42.967 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:42.974 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:42.974 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-19T18:51:43.028 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:43.028 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:43.063 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-19T18:51:43.122 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-19T18:51:43.169 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:43.188 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-19T18:51:43.272 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:43.387 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-19T18:51:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-19T18:51:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-19T18:51:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-19T18:51:43.553 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-19T18:51:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-19T18:51:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:51:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-19T18:51:43.729 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:43.746 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-19T18:51:43.789 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-19T18:51:43.823 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-19T18:51:43.918 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-19T18:51:43.954 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-19T18:51:44.030 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-19T18:51:44.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:44.045 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:44.045 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:44.046 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:44.046 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:44.046 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:44.173 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-19T18:51:44.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-02-19T18:51:44.304 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:51:44.466 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-02-19T18:51:44.500 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:51:44.524 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:51:44.524 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:44.524 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T18:51:44.525 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:51:44.525 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:51:44.525 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:44.858 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:58 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:45.030 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.0 M 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 12 M 2024-02-19T18:51:45.031 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:45.210 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-02-19T18:51:45.211 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:45.211 INFO:teuthology.orchestra.run.smithi086.stdout:Total 17 MB/s | 3.0 MB 00:00 2024-02-19T18:51:45.211 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:45.220 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:45.220 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:45.291 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:45.292 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:45.699 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:45.761 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:45.785 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:45.785 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T18:51:45.785 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:51:45.785 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:51:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:46.155 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:51:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:51:46.181 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:46.181 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T18:51:46.181 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:51:46.181 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:51:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:46.653 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:51:46.676 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:51:46.676 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:46.676 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T18:51:46.676 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:51:46.676 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:51:46.676 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:46.854 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:46.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:46.855 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:46.855 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:46.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:46.855 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:46.887 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:51:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:51:46.910 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:46.910 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T18:51:46.910 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:51:46.910 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:51:46.910 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:46.978 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:51:47.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-02-19T18:51:47.709 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:01 ago on Mon 19 Feb 2024 06:49:46 PM UTC. 2024-02-19T18:51:47.874 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-02-19T18:51:47.875 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-02-19T18:51:47.876 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-02-19T18:51:47.876 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:47.876 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 171 k 2024-02-19T18:51:47.876 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-02-19T18:51:47.876 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-02-19T18:51:47.946 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.4 MB/s | 171 kB 00:00 2024-02-19T18:51:47.947 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:47.947 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.3 MB/s | 171 kB 00:00 2024-02-19T18:51:47.947 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-02-19T18:51:47.954 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-02-19T18:51:47.955 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-02-19T18:51:48.009 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-02-19T18:51:48.010 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-02-19T18:51:48.160 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-02-19T18:51:48.252 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-19T18:51:48.550 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-19T18:51:48.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-19T18:51:48.552 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-19T18:51:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:49.054 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:49.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:49.055 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-02-19T18:51:49.055 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:49.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:51:49.055 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-02-19T18:51:49.291 DEBUG:teuthology.parallel:result is None 2024-02-19T18:51:49.580 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T18:51:49.581 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T18:51:49.582 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:49.583 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:49.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2024-02-19T18:51:50.492 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:51 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:50.610 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:50.664 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:50.665 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:51:50.665 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:50.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2024-02-19T18:51:51.315 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:51.486 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:51.487 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:51.487 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:51:51.487 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:51.487 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:51:51.487 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 221 k 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 213 k 2024-02-19T18:51:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:51:51.602 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-18.2.1-0.el9.noarch.rpm 1.9 MB/s | 221 kB 00:00 2024-02-19T18:51:51.602 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:51.602 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.9 MB/s | 221 kB 00:00 2024-02-19T18:51:51.602 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:51:51.608 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:51:51.608 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:51:51.610 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:51:51.611 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:51:51.684 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:51:52.070 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:51:52.158 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:51:52.535 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:51:52.824 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:51:52.824 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:52.824 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:51:52.824 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:51:52.825 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:52.825 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:53.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T18:51:53.609 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:53.782 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:53.783 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:53.783 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-02-19T18:51:53.783 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:53.783 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 142 k 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 395 k 2024-02-19T18:51:53.784 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:51:53.866 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.7 MB/s | 142 kB 00:00 2024-02-19T18:51:53.866 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:53.867 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.7 MB/s | 142 kB 00:00 2024-02-19T18:51:53.867 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:51:53.875 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:51:53.875 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:51:53.928 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:51:53.928 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:51:54.071 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:51:54.142 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:51:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T18:51:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:51:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:55.133 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:51:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-02-19T18:51:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:55.134 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:55.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2024-02-19T18:51:55.984 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:56.102 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:51:56.157 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:56.158 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:51:56.158 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:56.213 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T18:51:56.805 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:57 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:51:56.984 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:51:56.985 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:56.985 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:51:56.985 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:56.985 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:51:56.985 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-02-19T18:51:56.985 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout:Install 7 Packages 2024-02-19T18:51:56.986 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:56.987 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.1 M 2024-02-19T18:51:56.987 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 8.5 M 2024-02-19T18:51:56.987 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:51:57.145 INFO:teuthology.orchestra.run.smithi039.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 212 kB/s | 23 kB 00:00 2024-02-19T18:51:57.170 INFO:teuthology.orchestra.run.smithi039.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 109 kB/s | 15 kB 00:00 2024-02-19T18:51:57.220 INFO:teuthology.orchestra.run.smithi039.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.5 MB/s | 1.8 MB 00:00 2024-02-19T18:51:57.337 INFO:teuthology.orchestra.run.smithi039.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2024-02-19T18:51:57.404 INFO:teuthology.orchestra.run.smithi039.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 167 kB/s | 31 kB 00:00 2024-02-19T18:51:57.438 INFO:teuthology.orchestra.run.smithi039.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 153 kB/s | 41 kB 00:00 2024-02-19T18:51:57.588 INFO:teuthology.orchestra.run.smithi039.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 753 kB/s | 188 kB 00:00 2024-02-19T18:51:57.590 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:57.590 INFO:teuthology.orchestra.run.smithi039.stdout:Total 3.4 MB/s | 2.1 MB 00:00 2024-02-19T18:51:57.590 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:51:57.599 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:51:57.599 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:51:57.670 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:51:57.671 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:51:57.979 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:51:58.048 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-19T18:51:58.082 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-19T18:51:58.138 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-19T18:51:58.168 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-19T18:51:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-02-19T18:51:58.472 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-02-19T18:51:58.509 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-19T18:51:59.087 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:51:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:51:59.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T18:52:00.222 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:01 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:00.397 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:52:00.399 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:00.400 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:00.401 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2024-02-19T18:52:00.401 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:00.401 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 37 M 2024-02-19T18:52:00.401 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 209 M 2024-02-19T18:52:00.401 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:00.778 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 35 MB/s | 7.4 MB 00:00 2024-02-19T18:52:00.903 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 95 kB/s | 32 kB 00:00 2024-02-19T18:52:00.920 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 123 kB/s | 17 kB 00:00 2024-02-19T18:52:00.987 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 410 kB/s | 34 kB 00:00 2024-02-19T18:52:01.421 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.5 MB/s | 3.0 MB 00:00 2024-02-19T18:52:01.505 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 8.1 MB/s | 4.7 MB 00:00 2024-02-19T18:52:01.571 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.2 MB/s | 4.8 MB 00:00 2024-02-19T18:52:01.596 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 217 kB 00:00 2024-02-19T18:52:02.497 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-02-19T18:52:02.530 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 861 kB/s | 802 kB 00:00 2024-02-19T18:52:02.547 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 205 kB/s | 194 kB 00:00 2024-02-19T18:52:02.681 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-02-19T18:52:02.684 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:02.684 INFO:teuthology.orchestra.run.smithi039.stdout:Total 16 MB/s | 37 MB 00:02 2024-02-19T18:52:02.704 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:02.733 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:02.733 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:03.175 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:03.175 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:03.906 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:03.988 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-19T18:52:04.018 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-19T18:52:04.116 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:52:04.156 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-19T18:52:04.340 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:52:04.371 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:52:04.454 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-19T18:52:05.148 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-19T18:52:05.261 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-19T18:52:06.570 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-19T18:52:07.646 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-19T18:52:07.670 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-19T18:52:10.122 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T18:52:10.562 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T18:52:10.563 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T18:52:10.563 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T18:52:10.563 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T18:52:10.563 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T18:52:10.563 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:10.563 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:10.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2024-02-19T18:52:11.547 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:12 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:11.723 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:11.724 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:11.724 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:52:11.724 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:11.724 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:11.724 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-02-19T18:52:11.724 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:11.725 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:11.726 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:11.726 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2024-02-19T18:52:11.726 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:11.726 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.0 M 2024-02-19T18:52:11.726 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 26 M 2024-02-19T18:52:11.726 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:12.203 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 436 kB/s | 50 kB 00:00 2024-02-19T18:52:12.286 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 123 kB/s | 24 kB 00:00 2024-02-19T18:52:12.303 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 84 kB/s | 18 kB 00:00 2024-02-19T18:52:12.436 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 750 kB/s | 174 kB 00:00 2024-02-19T18:52:12.528 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 546 kB/s | 132 kB 00:00 2024-02-19T18:52:12.554 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 841 kB/s | 211 kB 00:00 2024-02-19T18:52:12.579 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 301 kB/s | 43 kB 00:00 2024-02-19T18:52:13.079 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 58 kB/s | 32 kB 00:00 2024-02-19T18:52:13.271 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): python3-google-auth-2.27.0-1.el9.noarch 293 kB/s | 203 kB 00:00 2024-02-19T18:52:13.430 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-02-19T18:52:13.455 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-certifi-2021.10.8-2.el9.noarch 17 kB/s | 15 kB 00:00 2024-02-19T18:52:13.530 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-02-19T18:52:13.647 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 413 kB/s | 90 kB 00:00 2024-02-19T18:52:13.652 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:13.652 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-02-19T18:52:13.652 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:13.668 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:13.668 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:13.779 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:13.780 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:14.304 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:14.512 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-19T18:52:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-19T18:52:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-19T18:52:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-19T18:52:14.708 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-19T18:52:14.807 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-19T18:52:14.915 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-19T18:52:14.969 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-19T18:52:15.745 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-19T18:52:15.797 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-19T18:52:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-19T18:52:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-19T18:52:15.909 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-19T18:52:16.662 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-19T18:52:16.663 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-19T18:52:16.663 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-19T18:52:16.663 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-19T18:52:16.663 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-19T18:52:17.041 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T18:52:17.042 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T18:52:17.043 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T18:52:17.043 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T18:52:17.043 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T18:52:17.043 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T18:52:17.043 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:17.280 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T18:52:17.871 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:18 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:18.043 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:18.044 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:18.044 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-02-19T18:52:18.044 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:18.044 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:18.044 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2024-02-19T18:52:18.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:18.046 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 6.8 M 2024-02-19T18:52:18.046 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 33 M 2024-02-19T18:52:18.046 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:18.432 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-02-19T18:52:18.649 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 705 kB/s | 227 kB 00:00 2024-02-19T18:52:18.674 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-02-19T18:52:18.733 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 878 kB/s | 51 kB 00:00 2024-02-19T18:52:18.800 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 383 kB/s | 58 kB 00:00 2024-02-19T18:52:18.975 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.9 MB/s | 5.8 MB 00:00 2024-02-19T18:52:18.977 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:18.977 INFO:teuthology.orchestra.run.smithi039.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-02-19T18:52:18.978 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:18.987 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:18.987 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:19.080 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:19.080 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:19.388 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:19.547 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-19T18:52:19.594 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-19T18:52:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-19T18:52:20.232 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-19T18:52:20.300 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-19T18:52:20.322 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:52:21.133 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:52:21.134 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-02-19T18:52:21.134 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-19T18:52:21.134 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-19T18:52:21.134 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-19T18:52:21.134 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:21.457 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:21.709 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2024-02-19T18:52:22.303 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:23 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:22.479 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:22.479 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2024-02-19T18:52:22.480 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:22.481 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 918 k 2024-02-19T18:52:22.481 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 2.7 M 2024-02-19T18:52:22.481 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:22.585 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 7.9 MB/s | 839 kB 00:00 2024-02-19T18:52:22.693 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 370 kB/s | 79 kB 00:00 2024-02-19T18:52:22.695 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:22.695 INFO:teuthology.orchestra.run.smithi039.stdout:Total 4.2 MB/s | 918 kB 00:00 2024-02-19T18:52:22.695 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:22.716 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:22.716 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:22.796 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:22.796 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:22.986 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:23.120 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-19T18:52:23.187 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:52:23.889 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:52:23.889 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-02-19T18:52:24.213 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-19T18:52:24.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:24.214 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:24.214 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-19T18:52:24.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:24.214 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:24.468 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2024-02-19T18:52:25.059 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:26 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:25.231 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:25.231 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:25.231 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:52:25.232 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:25.233 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 127 k 2024-02-19T18:52:25.233 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 456 k 2024-02-19T18:52:25.233 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:25.306 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.7 MB/s | 127 kB 00:00 2024-02-19T18:52:25.306 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:25.307 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.7 MB/s | 127 kB 00:00 2024-02-19T18:52:25.307 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:25.312 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:25.313 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:25.370 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:25.370 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:25.514 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:26.047 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:26.322 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:26.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:26.322 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:26.322 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:52:26.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:26.322 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:26.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2024-02-19T18:52:27.226 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:28 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:27.344 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:27.399 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:27.400 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:27.400 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:27.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2024-02-19T18:52:28.054 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:28.228 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:28.229 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:52:28.229 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:28.229 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 31 k 2024-02-19T18:52:28.229 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 139 k 2024-02-19T18:52:28.229 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:28.292 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 479 kB/s | 31 kB 00:00 2024-02-19T18:52:28.293 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:28.293 INFO:teuthology.orchestra.run.smithi039.stdout:Total 470 kB/s | 31 kB 00:00 2024-02-19T18:52:28.293 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:28.297 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:28.297 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:28.325 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:28.325 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:28.411 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:28.535 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:28.918 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:29.205 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:29.205 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:29.205 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:29.205 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:52:29.205 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:29.205 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:29.370 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2024-02-19T18:52:29.961 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:30.080 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:30.135 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:30.136 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:30.200 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2024-02-19T18:52:30.792 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:30.912 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:30.968 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:30.969 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:30.969 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:31.027 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2024-02-19T18:52:31.613 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:31.731 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:31.787 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:31.788 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:31.789 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:31.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2024-02-19T18:52:32.439 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:32.561 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:32.618 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:32.620 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:32.620 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:32.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2024-02-19T18:52:33.274 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:33.393 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:33.448 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:33.449 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:33.449 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:33.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2024-02-19T18:52:34.106 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:34.224 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:52:34.278 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:34.279 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-02-19T18:52:34.279 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:34.342 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2024-02-19T18:52:34.935 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:35.107 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:35.108 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:35.108 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:52:35.108 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 86 k 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 226 k 2024-02-19T18:52:35.109 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:35.179 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-02-19T18:52:35.179 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:35.180 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-02-19T18:52:35.180 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:35.187 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:35.187 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:35.245 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:35.245 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:35.500 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:35.605 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:36.150 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:36.446 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:36.446 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:36.446 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:36.446 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-02-19T18:52:36.446 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:36.446 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:36.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2024-02-19T18:52:37.259 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:37.434 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:52:37.436 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:37.436 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.0 M 2024-02-19T18:52:37.436 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 12 M 2024-02-19T18:52:37.436 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:37.550 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 26 MB/s | 3.0 MB 00:00 2024-02-19T18:52:37.551 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:37.551 INFO:teuthology.orchestra.run.smithi039.stdout:Total 26 MB/s | 3.0 MB 00:00 2024-02-19T18:52:37.551 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:37.561 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:37.561 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:37.637 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:37.638 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:38.070 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:38.118 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:38.141 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:38.141 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:52:38.142 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T18:52:38.142 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:52:38.142 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:52:38.142 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:39.302 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:39.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:39.302 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:39.302 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-02-19T18:52:39.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:39.302 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:39.553 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2024-02-19T18:52:40.152 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Feb 2024 06:50:59 PM UTC. 2024-02-19T18:52:40.326 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-02-19T18:52:40.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:40.327 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-02-19T18:52:40.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:40.327 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-02-19T18:52:40.327 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 171 k 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 494 k 2024-02-19T18:52:40.328 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-02-19T18:52:40.403 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.2 MB/s | 171 kB 00:00 2024-02-19T18:52:40.404 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:52:40.404 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.2 MB/s | 171 kB 00:00 2024-02-19T18:52:40.404 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-02-19T18:52:40.412 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-02-19T18:52:40.412 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-02-19T18:52:40.469 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-02-19T18:52:40.470 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-02-19T18:52:40.616 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-02-19T18:52:40.720 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:41.125 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:41.410 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:52:41.410 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:41.410 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-02-19T18:52:41.411 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:52:41.411 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:52:41.411 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-02-19T18:52:41.650 DEBUG:teuthology.parallel:result is None 2024-02-19T18:52:41.650 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:52:41.651 INFO:teuthology.packaging:ref: None 2024-02-19T18:52:41.651 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:52:41.651 INFO:teuthology.packaging:branch: None 2024-02-19T18:52:41.651 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:52:41.651 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:52:41.651 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:52:42.035 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T18:52:42.055 INFO:teuthology.orchestra.run.smithi039.stdout:18.2.1-0.el9 2024-02-19T18:52:42.056 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-02-19T18:52:42.056 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-02-19T18:52:42.058 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:52:42.058 INFO:teuthology.packaging:ref: None 2024-02-19T18:52:42.058 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:52:42.058 INFO:teuthology.packaging:branch: None 2024-02-19T18:52:42.058 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:52:42.058 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:52:42.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:52:42.196 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T18:52:42.221 INFO:teuthology.orchestra.run.smithi086.stdout:18.2.1-0.el9 2024-02-19T18:52:42.221 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-02-19T18:52:42.221 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-02-19T18:52:42.223 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T18:52:42.223 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:52:42.223 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T18:52:42.252 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:52:42.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T18:52:42.293 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T18:52:42.293 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:52:42.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T18:52:42.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T18:52:42.388 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:52:42.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T18:52:42.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T18:52:42.487 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T18:52:42.488 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:52:42.488 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T18:52:42.515 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T18:52:42.582 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:52:42.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T18:52:42.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T18:52:42.676 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T18:52:42.676 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:52:42.676 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T18:52:42.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T18:52:42.768 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:52:42.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T18:52:42.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T18:52:42.863 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:52:42.872 INFO:teuthology.task.print:**** done install task... 2024-02-19T18:52:42.872 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T18:52:43.010 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-19T18:52:43.010 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-02-19T18:52:43.010 INFO:tasks.cephadm:Cluster fsid is 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:52:43.011 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T18:52:43.011 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T18:52:43.011 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi039': '172.21.15.39', 'mon.smithi086': '172.21.15.86'} 2024-02-19T18:52:43.011 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T18:52:43.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname $(hostname -s) 2024-02-19T18:52:43.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-02-19T18:52:43.065 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-02-19T18:52:43.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:52:43.461 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76790/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1450-gf78a58c0', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1450.gf78a58c0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-15 19:25:41.455235', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240215.160715', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76792/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1450-gf78a58c0', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1450.gf78a58c0'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-15 19:17:06.436952', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240215.160715', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-19T18:52:44.082 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref reef, sha1 bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-02-19T18:52:44.083 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/reef/bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320/centos/9/x86_64/flavors/default/cephadm 2024-02-19T18:52:44.083 DEBUG:teuthology.orchestra.run.smithi039:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T18:52:49.645 INFO:teuthology.orchestra.run.smithi039.stdout:-rw-r--r--. 1 ubuntu ubuntu 214738 Feb 19 18:52 /home/ubuntu/cephtest/cephadm 2024-02-19T18:52:49.646 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T18:52:50.043 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 214738 Feb 19 18:52 /home/ubuntu/cephtest/cephadm 2024-02-19T18:52:50.044 DEBUG:teuthology.orchestra.run.smithi039:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T18:52:50.063 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T18:52:50.092 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-02-19T18:52:50.093 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-02-19T18:52:50.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-02-19T18:52:50.892 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:52:51.021 INFO:teuthology.orchestra.run.smithi039.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "image_id": "94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f", 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "repo_digests": [ 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0", 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "quay.io/ceph/ceph@sha256:e540726c37a4335a713579a0c9bf4413cb31ad156d22ea1ec00ed9bbc4317f66" 2024-02-19T18:53:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-02-19T18:53:12.807 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T18:54:33.790 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-02-19T18:54:33.790 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-02-19T18:54:33.790 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f", 2024-02-19T18:54:33.790 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-02-19T18:54:33.790 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0", 2024-02-19T18:54:33.791 INFO:teuthology.orchestra.run.smithi086.stdout: "quay.io/ceph/ceph@sha256:e540726c37a4335a713579a0c9bf4413cb31ad156d22ea1ec00ed9bbc4317f66" 2024-02-19T18:54:33.791 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-02-19T18:54:33.791 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-02-19T18:54:33.808 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph 2024-02-19T18:54:33.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-02-19T18:54:33.873 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /etc/ceph 2024-02-19T18:54:33.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-02-19T18:54:33.939 INFO:tasks.cephadm:Writing seed config... 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T18:54:33.941 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T18:54:33.942 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T18:54:33.943 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T18:54:33.944 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:54:33.944 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T18:54:33.968 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1026b4de-cf58-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T18:54:33.969 DEBUG:teuthology.orchestra.run.smithi039:mon.smithi039> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service 2024-02-19T18:54:34.011 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T18:54:34.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.39 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:54:34.173 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:54:34.173 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', '1026b4de-cf58-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.39', '--skip-admin-label'] 2024-02-19T18:54:34.197 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:54:34.197 INFO:teuthology.orchestra.run.smithi039.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-19T18:54:34.198 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying podman|docker is present... 2024-02-19T18:54:34.221 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:54:34.221 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying lvm2 is present... 2024-02-19T18:54:34.221 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying time synchronization is in place... 2024-02-19T18:54:34.229 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-19T18:54:34.229 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T18:54:34.234 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-19T18:54:34.235 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-02-19T18:54:34.241 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2024-02-19T18:54:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2024-02-19T18:54:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2024-02-19T18:54:34.247 INFO:teuthology.orchestra.run.smithi039.stdout:Repeating the final host check... 2024-02-19T18:54:34.269 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:54:34.270 INFO:teuthology.orchestra.run.smithi039.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-19T18:54:34.270 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl is present 2024-02-19T18:54:34.270 INFO:teuthology.orchestra.run.smithi039.stdout:lvcreate is present 2024-02-19T18:54:34.276 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-19T18:54:34.276 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T18:54:34.282 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-19T18:54:34.282 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-02-19T18:54:34.288 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2024-02-19T18:54:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2024-02-19T18:54:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2024-02-19T18:54:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:Host looks OK 2024-02-19T18:54:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:Cluster fsid: 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:Acquiring lock 140461481789664 on /run/cephadm/1026b4de-cf58-11ee-95bb-87774f69a715.lock 2024-02-19T18:54:34.294 INFO:teuthology.orchestra.run.smithi039.stdout:Lock 140461481789664 acquired on /run/cephadm/1026b4de-cf58-11ee-95bb-87774f69a715.lock 2024-02-19T18:54:34.295 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 3300 ... 2024-02-19T18:54:34.295 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 6789 ... 2024-02-19T18:54:34.295 INFO:teuthology.orchestra.run.smithi039.stdout:Base mon IP(s) is [172.21.15.39:3300, 172.21.15.39:6789], mon addrv is [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-02-19T18:54:34.298 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.39 metric 100 2024-02-19T18:54:34.298 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.39 metric 100 2024-02-19T18:54:34.300 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T18:54:34.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-19T18:54:34.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-19T18:54:34.303 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T18:54:34.303 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-19T18:54:34.303 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-19T18:54:34.303 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-19T18:54:34.303 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-02-19T18:54:34.303 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-19T18:54:34.304 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2024-02-19T18:54:34.304 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2024-02-19T18:54:34.304 INFO:teuthology.orchestra.run.smithi039.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-19T18:54:34.304 INFO:teuthology.orchestra.run.smithi039.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T18:54:34.305 INFO:teuthology.orchestra.run.smithi039.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Getting image source signatures 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:6345d49baed062606681bf114c10cebf2c69e90ec4d758270fae85ad0cc5c86b 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying config sha256:94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f 2024-02-19T18:54:35.553 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-19T18:54:36.581 INFO:teuthology.orchestra.run.smithi039.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-02-19T18:54:36.581 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-02-19T18:54:36.581 INFO:teuthology.orchestra.run.smithi039.stdout:Extracting ceph user uid/gid from container image... 2024-02-19T18:54:37.646 INFO:teuthology.orchestra.run.smithi039.stdout:stat: stdout 167 167 2024-02-19T18:54:37.646 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial keys... 2024-02-19T18:54:38.584 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQDto9Nlne/wOBAAoIo2a1cTu+VrPa4IYDuzsg== 2024-02-19T18:54:39.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQDuo9Nl0cLUNxAAOmETEZZMs4/BBuFAncHNYQ== 2024-02-19T18:54:40.579 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQDwo9NlTjoNARAAkM+vt0wLrCeKDFOD5er9Eg== 2024-02-19T18:54:40.580 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial monmap... 2024-02-19T18:54:41.576 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T18:54:41.576 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-02-19T18:54:41.576 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool for smithi039 [v2:172.21.15.39:3300,v1:172.21.15.39:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = pacific 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: set fsid to 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:54:41.577 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mon... 2024-02-19T18:54:42.424 INFO:teuthology.orchestra.run.smithi039.stdout:create mon.smithi039 on 2024-02-19T18:54:42.630 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-19T18:54:42.765 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T18:54:43.009 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1026b4de-cf58-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-1026b4de-cf58-11ee-95bb-87774f69a715.target. 2024-02-19T18:54:43.009 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1026b4de-cf58-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-1026b4de-cf58-11ee-95bb-87774f69a715.target. 2024-02-19T18:54:43.223 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039 2024-02-19T18:54:43.223 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service: Unit ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service not loaded. 2024-02-19T18:54:43.395 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1026b4de-cf58-11ee-95bb-87774f69a715.target.wants/ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service → /etc/systemd/system/ceph-1026b4de-cf58-11ee-95bb-87774f69a715@.service. 2024-02-19T18:54:43.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 podman[28588]: 2024-02-19 18:54:43.635018977 +0000 UTC m=+0.125993000 container create 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, org.label-schema.build-date=20240214, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1) 2024-02-19T18:54:43.945 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:54:43.946 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-02-19T18:54:43.953 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:54:43.953 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-02-19T18:54:43.953 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-02-19T18:54:43.953 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2024-02-19T18:54:43.953 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon to start... 2024-02-19T18:54:43.953 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon... 2024-02-19T18:54:44.071 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 podman[28588]: 2024-02-19 18:54:43.862829174 +0000 UTC m=+0.353803154 container init 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-19T18:54:44.072 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 podman[28588]: 2024-02-19 18:54:43.866467669 +0000 UTC m=+0.357441642 container start 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD) 2024-02-19T18:54:44.072 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 bash[28588]: 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac 2024-02-19T18:54:44.072 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 systemd[1]: Started Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T18:54:44.072 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 ceph-mon[28617]: mkfs 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:44.072 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:43 smithi039 ceph-mon[28617]: mon.smithi039 is new leader, mons smithi039 in quorum (ranks 0) 2024-02-19T18:54:45.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:44 smithi039 ceph-mon[28617]: mon.smithi039 is new leader, mons smithi039 in quorum (ranks 0) 2024-02-19T18:54:45.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:44 smithi039 ceph-mon[28617]: monmap e1: 1 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:54:45.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:44 smithi039 ceph-mon[28617]: fsmap 2024-02-19T18:54:45.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:44 smithi039 ceph-mon[28617]: osdmap e1: 0 total, 0 up, 0 in 2024-02-19T18:54:45.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:44 smithi039 ceph-mon[28617]: mgrmap e1: no daemons active 2024-02-19T18:54:45.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:44 smithi039 ceph-mon[28617]: from='client.? 172.21.15.39:0/2789224468' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T18:54:45.056 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout cluster: 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout id: 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout services: 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi039 (age 0.497134s) 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout data: 2024-02-19T18:54:45.062 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pgs: 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.350+0000 7f526ddf2700 1 Processor -- start 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.350+0000 7f526ddf2700 1 -- start start 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.350+0000 7f526ddf2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f5268106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.350+0000 7f526ddf2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268106e20 con 0x7f5268106430 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f5268106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f5268106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44962/0 (socket says 172.21.15.39:44962) 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52677fe700 1 -- 172.21.15.39:0/3227165183 learned_addr learned my addr 172.21.15.39:0/3227165183 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52677fe700 1 -- 172.21.15.39:0/3227165183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268107630 con 0x7f5268106430 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52677fe700 1 --2- 172.21.15.39:0/3227165183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f5268106850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f525c0098a0 tx=0x7f525c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e7912ca03c0f35 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:45.063 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52667fc700 1 -- 172.21.15.39:0/3227165183 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f525c004180 con 0x7f5268106430 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52667fc700 1 -- 172.21.15.39:0/3227165183 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c0042e0 con 0x7f5268106430 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.351+0000 7f52667fc700 1 -- 172.21.15.39:0/3227165183 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f525c0036a0 con 0x7f5268106430 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 -- 172.21.15.39:0/3227165183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 msgr2=0x7f5268106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 --2- 172.21.15.39:0/3227165183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f5268106850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f525c0098a0 tx=0x7f525c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 -- 172.21.15.39:0/3227165183 shutdown_connections 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 --2- 172.21.15.39:0/3227165183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f5268106850 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 -- 172.21.15.39:0/3227165183 >> 172.21.15.39:0/3227165183 conn(0x7f52681019b0 msgr2=0x7f5268103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 -- 172.21.15.39:0/3227165183 shutdown_connections 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.352+0000 7f526ddf2700 1 -- 172.21.15.39:0/3227165183 wait complete. 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f526ddf2700 1 Processor -- start 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f526ddf2700 1 -- start start 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f526ddf2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f526807ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f526ddf2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526807b090 con 0x7f5268106430 2024-02-19T18:54:45.064 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f52677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f526807ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f52677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f526807ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44966/0 (socket says 172.21.15.39:44966) 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f52677fe700 1 -- 172.21.15.39:0/2789224468 learned_addr learned my addr 172.21.15.39:0/2789224468 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.353+0000 7f52677fe700 1 -- 172.21.15.39:0/2789224468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c009550 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f52677fe700 1 --2- 172.21.15.39:0/2789224468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f526807ab50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f525c004000 tx=0x7f525c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f525c003ce0 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c020070 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52680791a0 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f525c011630 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5268079640 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f525c022030 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.354+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f525c025d00 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.355+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52680623c0 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.357+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f525c025480 con 0x7f5268106430 2024-02-19T18:54:45.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.430+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5268192cc0 con 0x7f5268106430 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.430+0000 7f5264ff9700 1 -- 172.21.15.39:0/2789224468 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f525c011c20 con 0x7f5268106430 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.431+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 msgr2=0x7f526807ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.431+0000 7f526ddf2700 1 --2- 172.21.15.39:0/2789224468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f526807ab50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f525c004000 tx=0x7f525c004750 comp rx=0 tx=0).stop 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.431+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 shutdown_connections 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.431+0000 7f526ddf2700 1 --2- 172.21.15.39:0/2789224468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5268106430 0x7f526807ab50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.432+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 >> 172.21.15.39:0/2789224468 conn(0x7f52681019b0 msgr2=0x7f526818ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.432+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 shutdown_connections 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:44.432+0000 7f526ddf2700 1 -- 172.21.15.39:0/2789224468 wait complete. 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:mon is available 2024-02-19T18:54:45.066 INFO:teuthology.orchestra.run.smithi039.stdout:Assimilating anything we can from ceph.conf... 2024-02-19T18:54:46.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:45 smithi039 ceph-mon[28617]: from='client.? 172.21.15.39:0/4288321134' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T18:54:46.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:45 smithi039 ceph-mon[28617]: from='client.? 172.21.15.39:0/4288321134' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T18:54:46.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:46.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2024-02-19T18:54:46.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:46.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-02-19T18:54:46.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-19T18:54:46.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.428+0000 7fcdd51ae700 1 Processor -- start 2024-02-19T18:54:46.253 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.428+0000 7fcdd51ae700 1 -- start start 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdd51ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdd51ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd0106e10 con 0x7fcdd0106420 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44978/0 (socket says 172.21.15.39:44978) 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdced9d700 1 -- 172.21.15.39:0/4052291513 learned_addr learned my addr 172.21.15.39:0/4052291513 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdced9d700 1 -- 172.21.15.39:0/4052291513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0107620 con 0x7fcdd0106420 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdced9d700 1 --2- 172.21.15.39:0/4052291513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0106840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40098a0 tx=0x7fcdc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619fc7937b51d785 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdcdd9b700 1 -- 172.21.15.39:0/4052291513 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4004180 con 0x7fcdd0106420 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdcdd9b700 1 -- 172.21.15.39:0/4052291513 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc40042e0 con 0x7fcdd0106420 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.429+0000 7fcdcdd9b700 1 -- 172.21.15.39:0/4052291513 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc40036a0 con 0x7fcdd0106420 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4052291513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 msgr2=0x7fcdd0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 --2- 172.21.15.39:0/4052291513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0106840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcdc40098a0 tx=0x7fcdc4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4052291513 shutdown_connections 2024-02-19T18:54:46.254 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 --2- 172.21.15.39:0/4052291513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0106840 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4052291513 >> 172.21.15.39:0/4052291513 conn(0x7fcdd0101a00 msgr2=0x7fcdd0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4052291513 shutdown_connections 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.430+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4052291513 wait complete. 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdd51ae700 1 Processor -- start 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdd51ae700 1 -- start start 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdd51ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdd51ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd0199dc0 con 0x7fcdd0106420 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44980/0 (socket says 172.21.15.39:44980) 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdced9d700 1 -- 172.21.15.39:0/4288321134 learned_addr learned my addr 172.21.15.39:0/4288321134 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.431+0000 7fcdced9d700 1 -- 172.21.15.39:0/4288321134 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4009550 con 0x7fcdd0106420 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdced9d700 1 --2- 172.21.15.39:0/4288321134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0199880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4004000 tx=0x7fcdc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:46.255 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4003ce0 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4020070 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdc4011630 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd0199fc0 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd019a3a0 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcdc4022030 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.432+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcdc4025d40 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.433+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd0193160 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.435+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fcdc4025480 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.504+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcdd002d040 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.509+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fcdc4011bd0 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.509+0000 7fcdb7fff700 1 -- 172.21.15.39:0/4288321134 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fcdc4036020 con 0x7fcdd0106420 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 msgr2=0x7fcdd0199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:46.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 --2- 172.21.15.39:0/4288321134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0199880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4004000 tx=0x7fcdc4004750 comp rx=0 tx=0).stop 2024-02-19T18:54:46.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 shutdown_connections 2024-02-19T18:54:46.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 --2- 172.21.15.39:0/4288321134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdd0106420 0x7fcdd0199880 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 >> 172.21.15.39:0/4288321134 conn(0x7fcdd0101a00 msgr2=0x7fcdd018dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:46.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 shutdown_connections 2024-02-19T18:54:46.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:45.510+0000 7fcdd51ae700 1 -- 172.21.15.39:0/4288321134 wait complete. 2024-02-19T18:54:46.257 INFO:teuthology.orchestra.run.smithi039.stdout:Generating new minimal ceph.conf... 2024-02-19T18:54:47.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:46 smithi039 ceph-mon[28617]: from='client.? 172.21.15.39:0/1215711143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.631+0000 7fdbb90e9700 1 Processor -- start 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.631+0000 7fdbb90e9700 1 -- start start 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb90e9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb4106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb90e9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb4106e10 con 0x7fdbb4106420 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb4106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb4106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59096/0 (socket says 172.21.15.39:59096) 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb2d9d700 1 -- 172.21.15.39:0/3723640115 learned_addr learned my addr 172.21.15.39:0/3723640115 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb2d9d700 1 -- 172.21.15.39:0/3723640115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4107620 con 0x7fdbb4106420 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb2d9d700 1 --2- 172.21.15.39:0/3723640115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb4106840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009e80 tx=0x7fdbac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a4e45898850af9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb1d9b700 1 -- 172.21.15.39:0/3723640115 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbac00e070 con 0x7fdbb4106420 2024-02-19T18:54:47.249 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb1d9b700 1 -- 172.21.15.39:0/3723640115 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdbac003e90 con 0x7fdbb4106420 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.632+0000 7fdbb1d9b700 1 -- 172.21.15.39:0/3723640115 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbac004380 con 0x7fdbb4106420 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 -- 172.21.15.39:0/3723640115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 msgr2=0x7fdbb4106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 --2- 172.21.15.39:0/3723640115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb4106840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009e80 tx=0x7fdbac00b330 comp rx=0 tx=0).stop 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 -- 172.21.15.39:0/3723640115 shutdown_connections 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 --2- 172.21.15.39:0/3723640115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb4106840 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 -- 172.21.15.39:0/3723640115 >> 172.21.15.39:0/3723640115 conn(0x7fdbb4101a00 msgr2=0x7fdbb4103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 -- 172.21.15.39:0/3723640115 shutdown_connections 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.633+0000 7fdbb90e9700 1 -- 172.21.15.39:0/3723640115 wait complete. 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb90e9700 1 Processor -- start 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb90e9700 1 -- start start 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb90e9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb90e9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb419a5a0 con 0x7fdbb4106420 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59104/0 (socket says 172.21.15.39:59104) 2024-02-19T18:54:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb2d9d700 1 -- 172.21.15.39:0/1215711143 learned_addr learned my addr 172.21.15.39:0/1215711143 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.634+0000 7fdbb2d9d700 1 -- 172.21.15.39:0/1215711143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac009b30 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbb2d9d700 1 --2- 172.21.15.39:0/1215711143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb419a060 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b7b0 tx=0x7fdbac004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbac00e040 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb419a7a0 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fdbac003c70 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbac01b730 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb419aca0 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdbac019040 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.635+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdbac007730 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.636+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb40623c0 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.638+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fdbac029060 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.708+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdbb410a820 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.708+0000 7fdbabfff700 1 -- 172.21.15.39:0/1215711143 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fdbac01f070 con 0x7fdbb4106420 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 msgr2=0x7fdbb419a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:47.251 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 --2- 172.21.15.39:0/1215711143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb419a060 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b7b0 tx=0x7fdbac004430 comp rx=0 tx=0).stop 2024-02-19T18:54:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 shutdown_connections 2024-02-19T18:54:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 --2- 172.21.15.39:0/1215711143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4106420 0x7fdbb419a060 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 >> 172.21.15.39:0/1215711143 conn(0x7fdbb4101a00 msgr2=0x7fdbb4102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 shutdown_connections 2024-02-19T18:54:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:46.709+0000 7fdbb90e9700 1 -- 172.21.15.39:0/1215711143 wait complete. 2024-02-19T18:54:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:Restarting the monitor... 2024-02-19T18:54:47.493 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:47 smithi039 systemd[1]: Stopping Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T18:54:47.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[28611]: 2024-02-19T18:54:47.503+0000 7fc551902700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi039 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T18:54:47.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[28611]: 2024-02-19T18:54:47.503+0000 7fc551902700 -1 mon.smithi039@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T18:54:47.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:47 smithi039 podman[28862]: 2024-02-19 18:54:47.667615489 +0000 UTC m=+0.274613554 container died 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-19T18:54:48.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 podman[28862]: 2024-02-19 18:54:48.036993401 +0000 UTC m=+0.643991473 container cleanup 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, RELEASE=HEAD, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.29.1) 2024-02-19T18:54:48.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 bash[28862]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039 2024-02-19T18:54:48.640 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 podman[28876]: 2024-02-19 18:54:48.271400705 +0000 UTC m=+0.608851143 container remove 66bc8365dd6cdba9b5ac0634ee7472986ecb6d0e3e79fc27e23f6c4ce376beac (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , ceph=True) 2024-02-19T18:54:48.640 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service: Deactivated successfully. 2024-02-19T18:54:48.640 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 systemd[1]: Stopped Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T18:54:48.640 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 systemd[1]: Starting Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 podman[28969]: 2024-02-19 18:54:48.528141134 +0000 UTC m=+0.018352549 image pull 94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f quay.io/ceph/ceph:v18.2.1 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 podman[28969]: 2024-02-19 18:54:48.63065346 +0000 UTC m=+0.120864912 container create ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 podman[28969]: 2024-02-19 18:54:48.833141106 +0000 UTC m=+0.323352523 container init ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 podman[28969]: 2024-02-19 18:54:48.835860379 +0000 UTC m=+0.326071803 container start ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: pidfile_write: ignore empty --pid-file 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: load: jerasure load: lrc 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: RocksDB version: 7.9.2 2024-02-19T18:54:48.941 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Git sha 0 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Compile date 2023-12-11 22:07:34 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: DB SUMMARY 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: DB Session ID: 9CN0XSLTXH2K52YA6825 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: CURRENT file: CURRENT 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi039/store.db dir, Total Num: 1, files: 000008.sst 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi039/store.db: 000009.log size: 97488 ; 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.error_if_exists: 0 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.create_if_missing: 0 2024-02-19T18:54:48.942 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.paranoid_checks: 1 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.env: 0x55f4ca4ea720 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.info_log: 0x55f4cd4f72e0 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.statistics: (nil) 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.use_fsync: 0 2024-02-19T18:54:48.943 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_log_file_size: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.allow_fallocate: 1 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.use_direct_reads: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.db_log_dir: 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.wal_dir: 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T18:54:48.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.write_buffer_manager: 0x55f4cc768140 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T18:54:48.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.unordered_write: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.row_cache: None 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.wal_filter: None 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.two_write_queues: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.wal_compression: 0 2024-02-19T18:54:48.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.atomic_flush: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.log_readahead_size: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_background_jobs: 2 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_background_compactions: -1 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_subcompactions: 1 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T18:54:48.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_open_files: -1 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_background_flushes: -1 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Compression algorithms supported: 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kZSTD supported: 0 2024-02-19T18:54:48.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kXpressCompression supported: 0 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kZlibCompression supported: 1 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kSnappyCompression supported: 1 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kLZ4Compression supported: 1 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: kBZip2Compression supported: 0 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi039/store.db/MANIFEST-000010 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.merge_operator: 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_filter: None 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T18:54:48.949 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f4cd4f73e0) 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: cache_index_and_filter_blocks: 1 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: pin_top_level_index_and_filter: 1 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: index_type: 0 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: data_block_index_type: 0 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: index_shortening: 1 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: checksum: 4 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: no_block_cache: 0 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache: 0x55f4cc809350 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache_name: BinnedLRUCache 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache_options: 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: capacity : 536870912 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: num_shard_bits : 4 2024-02-19T18:54:48.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: strict_capacity_limit : 0 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: high_pri_pool_ratio: 0.000 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache_compressed: (nil) 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: persistent_cache: (nil) 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_size: 4096 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_size_deviation: 10 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_restart_interval: 16 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: index_block_restart_interval: 1 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: metadata_block_size: 4096 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: partition_filters: 0 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: use_delta_encoding: 1 2024-02-19T18:54:48.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: filter_policy: bloomfilter 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: whole_key_filtering: 1 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: verify_compression: 0 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: read_amp_bytes_per_bit: 0 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: format_version: 5 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: enable_index_compression: 1 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_align: 0 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: max_auto_readahead_size: 262144 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: prepopulate_block_cache: 0 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: initial_auto_readahead_size: 8192 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression: NoCompression 2024-02-19T18:54:48.952 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.num_levels: 7 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T18:54:48.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T18:54:48.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T18:54:48.954 INFO:teuthology.orchestra.run.smithi039.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T18:54:48.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T18:54:48.956 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.table_properties_collectors: 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.inplace_update_support: 0 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T18:54:48.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.bloom_locality: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.max_successive_merges: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.ttl: 2592000 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.enable_blob_files: false 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.min_blob_size: 0 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T18:54:48.958 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi039/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fa20b246-dfe6-4930-93e3-91065e90de6b 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368888864370, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368888865879, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92838, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 90855, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83570, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708368888, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fa20b246-dfe6-4930-93e3-91065e90de6b", "db_session_id": "9CN0XSLTXH2K52YA6825", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368888865996, "job": 1, "event": "recovery_finished"} 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi039/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f4cc8a6000 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: DB pointer 0x55f4cc892000 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T18:54:48.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** DB Stats ** 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** Compaction Stats [default] ** 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: L0 2/0 92.54 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Sum 2/0 92.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:54:48.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 94.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** Compaction Stats [default] ** 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 94.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative compaction: 0.00 GB write, 13.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:54:48.961 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval compaction: 0.00 GB write, 13.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Block cache BinnedLRUCache@0x55f4cc809350#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: starting mon.smithi039 rank 0 at public addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] at bind addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi039 fsid 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???) e1 preinit fsid 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).mds e1 new map 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).mds e1 print_map 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: e1 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: legacy client fscid: -1 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: No filesystems configured 2024-02-19T18:54:48.962 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T18:54:48.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:54:48.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:54:48.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:54:48.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T18:54:48.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 bash[28969]: ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 2024-02-19T18:54:48.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 systemd[1]: Started Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T18:54:49.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mon.smithi039 is new leader, mons smithi039 in quorum (ranks 0) 2024-02-19T18:54:49.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: monmap e1: 1 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:54:49.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: fsmap 2024-02-19T18:54:49.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: osdmap e1: 0 total, 0 up, 0 in 2024-02-19T18:54:49.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:48 smithi039 ceph-mon[28988]: mgrmap e1: no daemons active 2024-02-19T18:54:49.983 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.340+0000 7fafe385b700 1 Processor -- start 2024-02-19T18:54:49.988 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.340+0000 7fafe385b700 1 -- start start 2024-02-19T18:54:49.988 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.340+0000 7fafe385b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc079220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:49.988 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.340+0000 7fafe385b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0797f0 con 0x7fafdc07adc0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.341+0000 7fafe15f7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc079220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.341+0000 7fafe15f7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc079220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59118/0 (socket says 172.21.15.39:59118) 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.341+0000 7fafe15f7700 1 -- 172.21.15.39:0/2203825224 learned_addr learned my addr 172.21.15.39:0/2203825224 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.341+0000 7fafe15f7700 1 -- 172.21.15.39:0/2203825224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc079930 con 0x7fafdc07adc0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.341+0000 7fafe15f7700 1 --2- 172.21.15.39:0/2203825224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc079220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fafd40098a0 tx=0x7fafd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a0667d8cd32fe1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafdbfff700 1 -- 172.21.15.39:0/2203825224 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafd4004180 con 0x7fafdc07adc0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafdbfff700 1 -- 172.21.15.39:0/2203825224 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fafd40042e0 con 0x7fafdc07adc0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafdbfff700 1 -- 172.21.15.39:0/2203825224 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafd4011a50 con 0x7fafdc07adc0 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 -- 172.21.15.39:0/2203825224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 msgr2=0x7fafdc079220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 --2- 172.21.15.39:0/2203825224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc079220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fafd40098a0 tx=0x7fafd4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 -- 172.21.15.39:0/2203825224 shutdown_connections 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 --2- 172.21.15.39:0/2203825224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc079220 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 -- 172.21.15.39:0/2203825224 >> 172.21.15.39:0/2203825224 conn(0x7fafdc1019b0 msgr2=0x7fafdc103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 -- 172.21.15.39:0/2203825224 shutdown_connections 2024-02-19T18:54:49.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.342+0000 7fafe385b700 1 -- 172.21.15.39:0/2203825224 wait complete. 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe385b700 1 Processor -- start 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe385b700 1 -- start start 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe385b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe385b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc19c850 con 0x7fafdc07adc0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe15f7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe15f7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc19c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59130/0 (socket says 172.21.15.39:59130) 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe15f7700 1 -- 172.21.15.39:0/4017928367 learned_addr learned my addr 172.21.15.39:0/4017928367 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.343+0000 7fafe15f7700 1 -- 172.21.15.39:0/4017928367 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd4009550 con 0x7fafdc07adc0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafe15f7700 1 --2- 172.21.15.39:0/4017928367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc19c310 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009520 tx=0x7fafd4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafd4003a80 con 0x7fafdc07adc0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fafd4003be0 con 0x7fafdc07adc0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc19ca50 con 0x7fafdc07adc0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafd40197b0 con 0x7fafdc07adc0 2024-02-19T18:54:49.990 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc19cef0 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fafd402d440 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.344+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fafd40247c0 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.345+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafdc0623c0 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.347+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fafd401ac30 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.416+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fafdc19fc40 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.418+0000 7fafda7fc700 1 -- 172.21.15.39:0/4017928367 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fafd4005a20 con 0x7fafdc07adc0 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 msgr2=0x7fafdc19c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 --2- 172.21.15.39:0/4017928367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc19c310 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009520 tx=0x7fafd4011ab0 comp rx=0 tx=0).stop 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 shutdown_connections 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 --2- 172.21.15.39:0/4017928367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafdc07adc0 0x7fafdc19c310 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 >> 172.21.15.39:0/4017928367 conn(0x7fafdc1019b0 msgr2=0x7fafdc103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 shutdown_connections 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:49.419+0000 7fafe385b700 1 -- 172.21.15.39:0/4017928367 wait complete. 2024-02-19T18:54:49.991 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-19T18:54:50.020 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:54:50.020 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mgr... 2024-02-19T18:54:50.020 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-19T18:54:50.021 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-19T18:54:50.021 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:8443 ... 2024-02-19T18:54:50.272 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mgr.smithi039.ydmpuh 2024-02-19T18:54:50.273 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mgr.smithi039.ydmpuh.service: Unit ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mgr.smithi039.ydmpuh.service not loaded. 2024-02-19T18:54:50.424 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1026b4de-cf58-11ee-95bb-87774f69a715.target.wants/ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mgr.smithi039.ydmpuh.service → /etc/systemd/system/ceph-1026b4de-cf58-11ee-95bb-87774f69a715@.service. 2024-02-19T18:54:50.650 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:50 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/4017928367' entity='client.admin' 2024-02-19T18:54:50.900 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:54:50.900 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-02-19T18:54:50.907 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:54:50.907 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-02-19T18:54:50.907 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-02-19T18:54:50.907 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2024-02-19T18:54:50.915 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:54:50.915 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-02-19T18:54:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:54:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-02-19T18:54:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-02-19T18:54:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-02-19T18:54:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr to start... 2024-02-19T18:54:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr... 2024-02-19T18:54:51.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:51 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2398626448' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1026b4de-cf58-11ee-95bb-87774f69a715", 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:54:52.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "smithi039" 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:54:52.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:52.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:54:52.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:54:52.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:54:52.107 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:54:52.107 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:54:52.108 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:54:52.109 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:54:43.947989+0000", 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.382+0000 7fd14cd39700 1 Processor -- start 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.382+0000 7fd14cd39700 1 -- start start 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.382+0000 7fd14cd39700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd148106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.382+0000 7fd14cd39700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd148106e20 con 0x7fd148106430 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd148106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd148106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59152/0 (socket says 172.21.15.39:59152) 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14659c700 1 -- 172.21.15.39:0/398415322 learned_addr learned my addr 172.21.15.39:0/398415322 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14659c700 1 -- 172.21.15.39:0/398415322 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148107630 con 0x7fd148106430 2024-02-19T18:54:52.110 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14659c700 1 --2- 172.21.15.39:0/398415322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd148106850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd1300098a0 tx=0x7fd130009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f0a89e1fdbf9f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14559a700 1 -- 172.21.15.39:0/398415322 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd130004180 con 0x7fd148106430 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14559a700 1 -- 172.21.15.39:0/398415322 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd1300042e0 con 0x7fd148106430 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.383+0000 7fd14559a700 1 -- 172.21.15.39:0/398415322 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd130011a50 con 0x7fd148106430 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 -- 172.21.15.39:0/398415322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 msgr2=0x7fd148106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 --2- 172.21.15.39:0/398415322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd148106850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd1300098a0 tx=0x7fd130009bb0 comp rx=0 tx=0).stop 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 -- 172.21.15.39:0/398415322 shutdown_connections 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 --2- 172.21.15.39:0/398415322 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd148106850 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 -- 172.21.15.39:0/398415322 >> 172.21.15.39:0/398415322 conn(0x7fd1481019d0 msgr2=0x7fd148103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 -- 172.21.15.39:0/398415322 shutdown_connections 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.384+0000 7fd14cd39700 1 -- 172.21.15.39:0/398415322 wait complete. 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14cd39700 1 Processor -- start 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14cd39700 1 -- start start 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14cd39700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd14819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14cd39700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14819a600 con 0x7fd148106430 2024-02-19T18:54:52.111 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd14819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd14819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59166/0 (socket says 172.21.15.39:59166) 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14659c700 1 -- 172.21.15.39:0/2398626448 learned_addr learned my addr 172.21.15.39:0/2398626448 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.385+0000 7fd14659c700 1 -- 172.21.15.39:0/2398626448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd130009550 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd14659c700 1 --2- 172.21.15.39:0/2398626448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd14819a0c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd130006af0 tx=0x7fd130011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd130003b90 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd13001a8a0 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14819a800 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd130019880 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14819aca0 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd13002d440 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.386+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd130024800 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.387+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1480623c0 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.389+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd13001aa10 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.461+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd14810a8e0 con 0x7fd148106430 2024-02-19T18:54:52.112 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.462+0000 7fd13f7fe700 1 -- 172.21.15.39:0/2398626448 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fd130035090 con 0x7fd148106430 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 msgr2=0x7fd14819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 --2- 172.21.15.39:0/2398626448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd14819a0c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd130006af0 tx=0x7fd130011a90 comp rx=0 tx=0).stop 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 shutdown_connections 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 --2- 172.21.15.39:0/2398626448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd148106430 0x7fd14819a0c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 >> 172.21.15.39:0/2398626448 conn(0x7fd1481019d0 msgr2=0x7fd148102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 shutdown_connections 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:51.463+0000 7fd14cd39700 1 -- 172.21.15.39:0/2398626448 wait complete. 2024-02-19T18:54:52.113 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (1/15)... 2024-02-19T18:54:55.199 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:54 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/908822386' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:54:55.221 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1026b4de-cf58-11ee-95bb-87774f69a715", 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:54:55.222 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "smithi039" 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:54:55.223 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:54:55.231 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:54:55.232 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:54:43.947989+0000", 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.533+0000 7ff013570700 1 Processor -- start 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.533+0000 7ff013570700 1 -- start start 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.533+0000 7ff013570700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c1050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.533+0000 7ff013570700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c074680 con 0x7ff00c104ca0 2024-02-19T18:54:55.233 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff01130c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c1050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff01130c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c1050c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59178/0 (socket says 172.21.15.39:59178) 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff01130c700 1 -- 172.21.15.39:0/1291936875 learned_addr learned my addr 172.21.15.39:0/1291936875 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff01130c700 1 -- 172.21.15.39:0/1291936875 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c105600 con 0x7ff00c104ca0 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff01130c700 1 --2- 172.21.15.39:0/1291936875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c1050c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0080098a0 tx=0x7ff008009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6ae2cf9cf67164 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff003fff700 1 -- 172.21.15.39:0/1291936875 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff008004180 con 0x7ff00c104ca0 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff003fff700 1 -- 172.21.15.39:0/1291936875 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff0080042e0 con 0x7ff00c104ca0 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.534+0000 7ff003fff700 1 -- 172.21.15.39:0/1291936875 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff008011a50 con 0x7ff00c104ca0 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 -- 172.21.15.39:0/1291936875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 msgr2=0x7ff00c1050c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 --2- 172.21.15.39:0/1291936875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c1050c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0080098a0 tx=0x7ff008009bb0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 -- 172.21.15.39:0/1291936875 shutdown_connections 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 --2- 172.21.15.39:0/1291936875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c1050c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 -- 172.21.15.39:0/1291936875 >> 172.21.15.39:0/1291936875 conn(0x7ff00c1008c0 msgr2=0x7ff00c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 -- 172.21.15.39:0/1291936875 shutdown_connections 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 -- 172.21.15.39:0/1291936875 wait complete. 2024-02-19T18:54:55.234 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.535+0000 7ff013570700 1 Processor -- start 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff013570700 1 -- start start 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff013570700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff013570700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c19c850 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff01130c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff01130c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c19c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59184/0 (socket says 172.21.15.39:59184) 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff01130c700 1 -- 172.21.15.39:0/908822386 learned_addr learned my addr 172.21.15.39:0/908822386 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff01130c700 1 -- 172.21.15.39:0/908822386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff008009550 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff01130c700 1 --2- 172.21.15.39:0/908822386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c19c310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff008003de0 tx=0x7ff008011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff008003930 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c19ca50 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.536+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff008003a90 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.537+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff008019730 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.537+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c19cef0 con 0x7ff00c104ca0 2024-02-19T18:54:55.235 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.537+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff008019d30 con 0x7ff00c104ca0 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.537+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff00800b520 con 0x7ff00c104ca0 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.537+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00c195bc0 con 0x7ff00c104ca0 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.540+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7ff00801abc0 con 0x7ff00c104ca0 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.612+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff00c02cd60 con 0x7ff00c104ca0 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.613+0000 7ff0027fc700 1 -- 172.21.15.39:0/908822386 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7ff00801a430 con 0x7ff00c104ca0 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 msgr2=0x7ff00c19c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 --2- 172.21.15.39:0/908822386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c19c310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff008003de0 tx=0x7ff008011ab0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 shutdown_connections 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 --2- 172.21.15.39:0/908822386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff00c104ca0 0x7ff00c19c310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 >> 172.21.15.39:0/908822386 conn(0x7ff00c1008c0 msgr2=0x7ff00c190860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 shutdown_connections 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:54.614+0000 7ff013570700 1 -- 172.21.15.39:0/908822386 wait complete. 2024-02-19T18:54:55.236 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (2/15)... 2024-02-19T18:54:57.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: Activating manager daemon smithi039.ydmpuh 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: mgrmap e2: smithi039.ydmpuh(active, starting, since 0.00469304s) 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:54:57.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:57 smithi039 ceph-mon[28988]: from='mgr.14100 172.21.15.39:0/2257864459' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:54:58.295 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:54:58.295 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1026b4de-cf58-11ee-95bb-87774f69a715", 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "smithi039" 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:58.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:54:58.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:54:58.305 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:54:58.306 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:54:43.947989+0000", 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bf575700 1 Processor -- start 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bf575700 1 -- start start 2024-02-19T18:54:58.307 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bf575700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b8106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bf575700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8106e20 con 0x7fb4b8106430 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bd311700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b8106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bd311700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b8106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43616/0 (socket says 172.21.15.39:43616) 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.636+0000 7fb4bd311700 1 -- 172.21.15.39:0/270060642 learned_addr learned my addr 172.21.15.39:0/270060642 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.637+0000 7fb4bd311700 1 -- 172.21.15.39:0/270060642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8107630 con 0x7fb4b8106430 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.637+0000 7fb4bd311700 1 --2- 172.21.15.39:0/270060642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b8106850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40098a0 tx=0x7fb4b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e5d4db1a5dee38 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.637+0000 7fb4affff700 1 -- 172.21.15.39:0/270060642 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b4004180 con 0x7fb4b8106430 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.639+0000 7fb4affff700 1 -- 172.21.15.39:0/270060642 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb4b40042e0 con 0x7fb4b8106430 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.639+0000 7fb4affff700 1 -- 172.21.15.39:0/270060642 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b4011a50 con 0x7fb4b8106430 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 -- 172.21.15.39:0/270060642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 msgr2=0x7fb4b8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 --2- 172.21.15.39:0/270060642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b8106850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40098a0 tx=0x7fb4b4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 -- 172.21.15.39:0/270060642 shutdown_connections 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 --2- 172.21.15.39:0/270060642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b8106850 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:58.308 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 -- 172.21.15.39:0/270060642 >> 172.21.15.39:0/270060642 conn(0x7fb4b81019b0 msgr2=0x7fb4b8103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 -- 172.21.15.39:0/270060642 shutdown_connections 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.640+0000 7fb4bf575700 1 -- 172.21.15.39:0/270060642 wait complete. 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bf575700 1 Processor -- start 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bf575700 1 -- start start 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bf575700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bf575700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b819a670 con 0x7fb4b8106430 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bd311700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bd311700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43628/0 (socket says 172.21.15.39:43628) 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.641+0000 7fb4bd311700 1 -- 172.21.15.39:0/228840939 learned_addr learned my addr 172.21.15.39:0/228840939 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4bd311700 1 -- 172.21.15.39:0/228840939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4009550 con 0x7fb4b8106430 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4bd311700 1 --2- 172.21.15.39:0/228840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b819a130 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009520 tx=0x7fb4b4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b4003a80 con 0x7fb4b8106430 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b819a870 con 0x7fb4b8106430 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb4b4003be0 con 0x7fb4b8106430 2024-02-19T18:54:58.309 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b40197b0 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.642+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b819ad10 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.643+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 2) v1 ==== 45039+0+0 (secure 0 0 0) 0x7fb4b4019a40 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.643+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb4b404cfc0 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.643+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c005320 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.646+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb4b4019cf0 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.722+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb49c0059f0 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.722+0000 7fb4ae7fc700 1 -- 172.21.15.39:0/228840939 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fb4b40172b0 con 0x7fb4b8106430 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 msgr2=0x7fb4b819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 --2- 172.21.15.39:0/228840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b819a130 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009520 tx=0x7fb4b4011ab0 comp rx=0 tx=0).stop 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 shutdown_connections 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 --2- 172.21.15.39:0/228840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4b8106430 0x7fb4b819a130 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 >> 172.21.15.39:0/228840939 conn(0x7fb4b81019b0 msgr2=0x7fb4b81030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 shutdown_connections 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:54:57.723+0000 7fb4bf575700 1 -- 172.21.15.39:0/228840939 wait complete. 2024-02-19T18:54:58.310 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (3/15)... 2024-02-19T18:54:58.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:58 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/228840939' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:54:58.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:54:58 smithi039 ceph-mon[28988]: mgrmap e3: smithi039.ydmpuh(active, since 1.01036s) 2024-02-19T18:55:00.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:00 smithi039 ceph-mon[28988]: mgrmap e4: smithi039.ydmpuh(active, since 2s) 2024-02-19T18:55:01.745 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:55:01.745 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:55:01.745 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1026b4de-cf58-11ee-95bb-87774f69a715", 2024-02-19T18:55:01.745 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "smithi039" 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-19T18:55:01.746 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:55:01.747 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:55:01.756 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:55:01.757 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:54:43.947989+0000", 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.715+0000 7f400efa4700 1 Processor -- start 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.715+0000 7f400efa4700 1 -- start start 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.715+0000 7f400efa4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f40081092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.715+0000 7f400efa4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4008074680 con 0x7f4008106eb0 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f400cd40700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f40081092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:01.758 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f400cd40700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f40081092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43682/0 (socket says 172.21.15.39:43682) 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f400cd40700 1 -- 172.21.15.39:0/3078829436 learned_addr learned my addr 172.21.15.39:0/3078829436 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f400cd40700 1 -- 172.21.15.39:0/3078829436 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40081097e0 con 0x7f4008106eb0 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f400cd40700 1 --2- 172.21.15.39:0/3078829436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f40081092a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0098a0 tx=0x7f3ffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b46ff06f70813b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f40077fe700 1 -- 172.21.15.39:0/3078829436 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc004180 con 0x7f4008106eb0 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f40077fe700 1 -- 172.21.15.39:0/3078829436 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3ffc0042e0 con 0x7f4008106eb0 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.716+0000 7f40077fe700 1 -- 172.21.15.39:0/3078829436 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc011a50 con 0x7f4008106eb0 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 -- 172.21.15.39:0/3078829436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 msgr2=0x7f40081092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 --2- 172.21.15.39:0/3078829436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f40081092a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0098a0 tx=0x7f3ffc009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 -- 172.21.15.39:0/3078829436 shutdown_connections 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 --2- 172.21.15.39:0/3078829436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f40081092a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 -- 172.21.15.39:0/3078829436 >> 172.21.15.39:0/3078829436 conn(0x7f40081008c0 msgr2=0x7f4008102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 -- 172.21.15.39:0/3078829436 shutdown_connections 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.717+0000 7f400efa4700 1 -- 172.21.15.39:0/3078829436 wait complete. 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400efa4700 1 Processor -- start 2024-02-19T18:55:01.759 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400efa4700 1 -- start start 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400efa4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f400819c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400efa4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400819c840 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400cd40700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f400819c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400cd40700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f400819c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43686/0 (socket says 172.21.15.39:43686) 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400cd40700 1 -- 172.21.15.39:0/845803462 learned_addr learned my addr 172.21.15.39:0/845803462 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400cd40700 1 -- 172.21.15.39:0/845803462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc009550 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f400cd40700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f400819c300 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc003de0 tx=0x7f3ffc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.718+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc003930 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.719+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400819ca40 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.719+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f3ffc003a90 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.719+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc019730 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.719+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400819cee0 con 0x7f4008106eb0 2024-02-19T18:55:01.760 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.720+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f3ffc0199d0 con 0x7f4008106eb0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.720+0000 7f4005ffb700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f3ff80382d0 0x7f3ff803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.720+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3ffc04c0b0 con 0x7f4008106eb0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.720+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400810a560 con 0x7f4008106eb0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.720+0000 7f4007fff700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f3ff80382d0 0x7f3ff803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.720+0000 7f4007fff700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f3ff80382d0 0x7f3ff803a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4006fd0 tx=0x7f3ff4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:00.725+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3ffc02c3a0 con 0x7f4008106eb0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.001+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f400810a780 con 0x7f4008106eb0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.001+0000 7f4005ffb700 1 -- 172.21.15.39:0/845803462 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f3ffc010380 con 0x7f4008106eb0 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f3ff80382d0 msgr2=0x7f3ff803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f3ff80382d0 0x7f3ff803a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4006fd0 tx=0x7f3ff4006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 msgr2=0x7f400819c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f400819c300 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc003de0 tx=0x7f3ffc011ab0 comp rx=0 tx=0).stop 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 shutdown_connections 2024-02-19T18:55:01.761 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f3ff80382d0 0x7f3ff803a790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:01.762 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 --2- 172.21.15.39:0/845803462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4008106eb0 0x7f400819c300 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:01.762 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.003+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 >> 172.21.15.39:0/845803462 conn(0x7f40081008c0 msgr2=0x7f4008101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:01.762 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.004+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 shutdown_connections 2024-02-19T18:55:01.762 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:01.004+0000 7f400efa4700 1 -- 172.21.15.39:0/845803462 wait complete. 2024-02-19T18:55:01.762 INFO:teuthology.orchestra.run.smithi039.stdout:mgr is available 2024-02-19T18:55:01.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:01 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/845803462' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:55:02.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:02 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2788329430' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T18:55:02.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:55:02.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2024-02-19T18:55:02.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:55:02.965 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-19T18:55:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8f505a700 1 Processor -- start 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8f505a700 1 -- start start 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8f505a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f0106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8f505a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0106e10 con 0x7fe8f0106420 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8eed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8eed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43690/0 (socket says 172.21.15.39:43690) 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8eed9d700 1 -- 172.21.15.39:0/3153600939 learned_addr learned my addr 172.21.15.39:0/3153600939 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.155+0000 7fe8eed9d700 1 -- 172.21.15.39:0/3153600939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0107620 con 0x7fe8f0106420 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8eed9d700 1 --2- 172.21.15.39:0/3153600939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f0106840 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00098a0 tx=0x7fe8e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9333fcfd277efaab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8edd9b700 1 -- 172.21.15.39:0/3153600939 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e0004180 con 0x7fe8f0106420 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8edd9b700 1 -- 172.21.15.39:0/3153600939 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8e00042e0 con 0x7fe8f0106420 2024-02-19T18:55:02.967 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8edd9b700 1 -- 172.21.15.39:0/3153600939 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e0011a50 con 0x7fe8f0106420 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8f505a700 1 -- 172.21.15.39:0/3153600939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 msgr2=0x7fe8f0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8f505a700 1 --2- 172.21.15.39:0/3153600939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f0106840 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00098a0 tx=0x7fe8e0009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8f505a700 1 -- 172.21.15.39:0/3153600939 shutdown_connections 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8f505a700 1 --2- 172.21.15.39:0/3153600939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f0106840 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.156+0000 7fe8f505a700 1 -- 172.21.15.39:0/3153600939 >> 172.21.15.39:0/3153600939 conn(0x7fe8f0101a00 msgr2=0x7fe8f0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.157+0000 7fe8f505a700 1 -- 172.21.15.39:0/3153600939 shutdown_connections 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.157+0000 7fe8f505a700 1 -- 172.21.15.39:0/3153600939 wait complete. 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.157+0000 7fe8f505a700 1 Processor -- start 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.157+0000 7fe8f505a700 1 -- start start 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8f505a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8f505a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f019a5f0 con 0x7fe8f0106420 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8eed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:02.968 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8eed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43704/0 (socket says 172.21.15.39:43704) 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8eed9d700 1 -- 172.21.15.39:0/2788329430 learned_addr learned my addr 172.21.15.39:0/2788329430 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8eed9d700 1 -- 172.21.15.39:0/2788329430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0009550 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8eed9d700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f019a0b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006af0 tx=0x7fe8e0011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e0003b90 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f019a7f0 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8e001a8a0 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.158+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e0019880 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.159+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f019ac90 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.159+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fe8e0019b40 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.159+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f00623c0 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.160+0000 7fe8d7fff700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fe8d8038290 0x7fe8d803a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.160+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe8e004c630 con 0x7fe8f0106420 2024-02-19T18:55:02.969 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.160+0000 7fe8ee59c700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fe8d8038290 0x7fe8d803a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.160+0000 7fe8ee59c700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fe8d8038290 0x7fe8d803a750 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4006fd0 tx=0x7fe8e4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.164+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe8e002b750 con 0x7fe8f0106420 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.362+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe8f010a820 con 0x7fe8f0106420 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.362+0000 7fe8d7fff700 1 -- 172.21.15.39:0/2788329430 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe8e00172b0 con 0x7fe8f0106420 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fe8d8038290 msgr2=0x7fe8d803a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fe8d8038290 0x7fe8d803a750 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4006fd0 tx=0x7fe8e4006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 msgr2=0x7fe8f019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f019a0b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006af0 tx=0x7fe8e0011a90 comp rx=0 tx=0).stop 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 shutdown_connections 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fe8d8038290 0x7fe8d803a750 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 --2- 172.21.15.39:0/2788329430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8f0106420 0x7fe8f019a0b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.365+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 >> 172.21.15.39:0/2788329430 conn(0x7fe8f0101a00 msgr2=0x7fe8f01027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.366+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 shutdown_connections 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:02.366+0000 7fe8f505a700 1 -- 172.21.15.39:0/2788329430 wait complete. 2024-02-19T18:55:02.970 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling cephadm module... 2024-02-19T18:55:04.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:03 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/25143763' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T18:55:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.369+0000 7f919d4f3700 1 Processor -- start 2024-02-19T18:55:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.369+0000 7f919d4f3700 1 -- start start 2024-02-19T18:55:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.369+0000 7f919d4f3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f91981092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.369+0000 7f919d4f3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198079e80 con 0x7f9198106f00 2024-02-19T18:55:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.369+0000 7f9196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f91981092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f91981092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43720/0 (socket says 172.21.15.39:43720) 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9196ffd700 1 -- 172.21.15.39:0/2709886968 learned_addr learned my addr 172.21.15.39:0/2709886968 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9196ffd700 1 -- 172.21.15.39:0/2709886968 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198109830 con 0x7f9198106f00 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9196ffd700 1 --2- 172.21.15.39:0/2709886968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f91981092f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9190009e80 tx=0x7f919000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b19695316781c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9195ffb700 1 -- 172.21.15.39:0/2709886968 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919000e070 con 0x7f9198106f00 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9195ffb700 1 -- 172.21.15.39:0/2709886968 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9190003e90 con 0x7f9198106f00 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.370+0000 7f9195ffb700 1 -- 172.21.15.39:0/2709886968 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9190004380 con 0x7f9198106f00 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 -- 172.21.15.39:0/2709886968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 msgr2=0x7f91981092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 --2- 172.21.15.39:0/2709886968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f91981092f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9190009e80 tx=0x7f919000b330 comp rx=0 tx=0).stop 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 -- 172.21.15.39:0/2709886968 shutdown_connections 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 --2- 172.21.15.39:0/2709886968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f91981092f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:04.580 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 -- 172.21.15.39:0/2709886968 >> 172.21.15.39:0/2709886968 conn(0x7f91981008d0 msgr2=0x7f9198102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 -- 172.21.15.39:0/2709886968 shutdown_connections 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 -- 172.21.15.39:0/2709886968 wait complete. 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.371+0000 7f919d4f3700 1 Processor -- start 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f919d4f3700 1 -- start start 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f919d4f3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f9198106860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f919d4f3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198106da0 con 0x7f9198106f00 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f9196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f9198106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f9196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f9198106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43734/0 (socket says 172.21.15.39:43734) 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f9196ffd700 1 -- 172.21.15.39:0/25143763 learned_addr learned my addr 172.21.15.39:0/25143763 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f9196ffd700 1 -- 172.21.15.39:0/25143763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190009b30 con 0x7f9198106f00 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f9196ffd700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f9198106860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f919000d010 tx=0x7f9190004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:04.581 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919000e040 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9190003c30 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9198104eb0 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919001b790 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.372+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91981053b0 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.373+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f9190019040 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.373+0000 7f918ffff700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f91840382e0 0x7f918403a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.373+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f919004bcf0 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.374+0000 7f91967fc700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f91840382e0 0x7f918403a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.374+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919810a580 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.374+0000 7f91967fc700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f91840382e0 0x7f918403a7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9180006fd0 tx=0x7f9180006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.379+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9190011980 con 0x7f9198106f00 2024-02-19T18:55:04.582 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.632+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f919810a770 con 0x7f9198106f00 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.903+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f919001a390 con 0x7f9198106f00 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.904+0000 7f918ffff700 1 -- 172.21.15.39:0/25143763 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9190021de0 con 0x7f9198106f00 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f91840382e0 msgr2=0x7f918403a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f91840382e0 0x7f918403a7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9180006fd0 tx=0x7f9180006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 msgr2=0x7f9198106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f9198106860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f919000d010 tx=0x7f9190004430 comp rx=0 tx=0).stop 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 shutdown_connections 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f91840382e0 0x7f918403a7a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 --2- 172.21.15.39:0/25143763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9198106f00 0x7f9198106860 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.906+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 >> 172.21.15.39:0/25143763 conn(0x7f91981008d0 msgr2=0x7f9198101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.907+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 shutdown_connections 2024-02-19T18:55:04.583 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:03.907+0000 7f919d4f3700 1 -- 172.21.15.39:0/25143763 wait complete. 2024-02-19T18:55:05.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:04 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/25143763' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T18:55:05.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:04 smithi039 ceph-mon[28988]: mgrmap e5: smithi039.ydmpuh(active, since 6s) 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "smithi039.ydmpuh", 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.955+0000 7fa9eb763700 1 Processor -- start 2024-02-19T18:55:05.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9eb763700 1 -- start start 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9eb763700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e4106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9eb763700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4106e20 con 0x7fa9e4106430 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9e94ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e4106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9e94ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e4106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43762/0 (socket says 172.21.15.39:43762) 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9e94ff700 1 -- 172.21.15.39:0/78255988 learned_addr learned my addr 172.21.15.39:0/78255988 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9e94ff700 1 -- 172.21.15.39:0/78255988 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e4107630 con 0x7fa9e4106430 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.956+0000 7fa9e94ff700 1 --2- 172.21.15.39:0/78255988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e4106850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00098a0 tx=0x7fa9e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f4852742d20d1e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9dbfff700 1 -- 172.21.15.39:0/78255988 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e0004180 con 0x7fa9e4106430 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9dbfff700 1 -- 172.21.15.39:0/78255988 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa9e00042e0 con 0x7fa9e4106430 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9dbfff700 1 -- 172.21.15.39:0/78255988 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e0011a50 con 0x7fa9e4106430 2024-02-19T18:55:05.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 -- 172.21.15.39:0/78255988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 msgr2=0x7fa9e4106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 --2- 172.21.15.39:0/78255988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e4106850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00098a0 tx=0x7fa9e0009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 -- 172.21.15.39:0/78255988 shutdown_connections 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 --2- 172.21.15.39:0/78255988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e4106850 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 -- 172.21.15.39:0/78255988 >> 172.21.15.39:0/78255988 conn(0x7fa9e41019d0 msgr2=0x7fa9e4103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 -- 172.21.15.39:0/78255988 shutdown_connections 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.957+0000 7fa9eb763700 1 -- 172.21.15.39:0/78255988 wait complete. 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9eb763700 1 Processor -- start 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9eb763700 1 -- start start 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9eb763700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9eb763700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e419a600 con 0x7fa9e4106430 2024-02-19T18:55:05.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9e94ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9e94ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43772/0 (socket says 172.21.15.39:43772) 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9e94ff700 1 -- 172.21.15.39:0/2680927269 learned_addr learned my addr 172.21.15.39:0/2680927269 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.958+0000 7fa9e94ff700 1 -- 172.21.15.39:0/2680927269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0009550 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.959+0000 7fa9e94ff700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e419a0c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0000c00 tx=0x7fa9e0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.959+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e0003b90 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.959+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa9e00248b0 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.959+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e419a800 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.959+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e001a8e0 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.959+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e419aca0 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.960+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa9e001aba0 con 0x7fa9e4106430 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.960+0000 7fa9da7fc700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 0x7fa9d003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:05.841 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.960+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa9e004c090 con 0x7fa9e4106430 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.960+0000 7fa9e8cfe700 1 -- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 msgr2=0x7fa9d003a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.960+0000 7fa9e8cfe700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 0x7fa9d003a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.960+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e40623c0 con 0x7fa9e4106430 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:04.965+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa9e00233a0 con 0x7fa9e4106430 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.161+0000 7fa9e8cfe700 1 -- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 msgr2=0x7fa9d003a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.161+0000 7fa9e8cfe700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 0x7fa9d003a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.222+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa9e419dcd0 con 0x7fa9e4106430 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.222+0000 7fa9da7fc700 1 -- 172.21.15.39:0/2680927269 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa9e0018ba0 con 0x7fa9e4106430 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 msgr2=0x7fa9d003a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 0x7fa9d003a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:05.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 msgr2=0x7fa9e419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e419a0c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0000c00 tx=0x7fa9e0011ab0 comp rx=0 tx=0).stop 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 shutdown_connections 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7fa9d0038320 0x7fa9d003a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 --2- 172.21.15.39:0/2680927269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e4106430 0x7fa9e419a0c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 >> 172.21.15.39:0/2680927269 conn(0x7fa9e41019d0 msgr2=0x7fa9e4102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 shutdown_connections 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:05.224+0000 7fa9eb763700 1 -- 172.21.15.39:0/2680927269 wait complete. 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2024-02-19T18:55:05.843 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 5... 2024-02-19T18:55:06.131 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:05 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2680927269' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T18:55:11.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: Active manager daemon smithi039.ydmpuh restarted 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: Activating manager daemon smithi039.ydmpuh 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: osdmap e2: 0 total, 0 up, 0 in 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: mgrmap e6: smithi039.ydmpuh(active, starting, since 0.00848812s) 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:11.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:10 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:12.125 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:11 smithi039 ceph-mon[28988]: Found migration_current of "None". Setting to last migration. 2024-02-19T18:55:12.125 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:11 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:12.126 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:11 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:55:12.126 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:11 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T18:55:12.126 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:11 smithi039 ceph-mon[28988]: mgrmap e7: smithi039.ydmpuh(active, since 1.01456s) 2024-02-19T18:55:12.445 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:55:12.445 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.242+0000 7f3501dc1700 1 Processor -- start 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.242+0000 7f3501dc1700 1 -- start start 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.243+0000 7f3501dc1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc0791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.243+0000 7f3501dc1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0797b0 con 0x7f34fc07ad80 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.243+0000 7f34fb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc0791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.243+0000 7f34fb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc0791e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43784/0 (socket says 172.21.15.39:43784) 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.243+0000 7f34fb7fe700 1 -- 172.21.15.39:0/4025282763 learned_addr learned my addr 172.21.15.39:0/4025282763 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.243+0000 7f34fb7fe700 1 -- 172.21.15.39:0/4025282763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0798f0 con 0x7f34fc07ad80 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f34fb7fe700 1 --2- 172.21.15.39:0/4025282763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc0791e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098a0 tx=0x7f34ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f439c3f4a7fc8ed3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:12.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f34fa7fc700 1 -- 172.21.15.39:0/4025282763 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ec004180 con 0x7f34fc07ad80 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f34fa7fc700 1 -- 172.21.15.39:0/4025282763 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f34ec0042e0 con 0x7f34fc07ad80 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f34fa7fc700 1 -- 172.21.15.39:0/4025282763 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ec011a50 con 0x7f34fc07ad80 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f3501dc1700 1 -- 172.21.15.39:0/4025282763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 msgr2=0x7f34fc0791e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f3501dc1700 1 --2- 172.21.15.39:0/4025282763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc0791e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f34ec0098a0 tx=0x7f34ec009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f3501dc1700 1 -- 172.21.15.39:0/4025282763 shutdown_connections 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f3501dc1700 1 --2- 172.21.15.39:0/4025282763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc0791e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.244+0000 7f3501dc1700 1 -- 172.21.15.39:0/4025282763 >> 172.21.15.39:0/4025282763 conn(0x7f34fc1019b0 msgr2=0x7f34fc103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.245+0000 7f3501dc1700 1 -- 172.21.15.39:0/4025282763 shutdown_connections 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.245+0000 7f3501dc1700 1 -- 172.21.15.39:0/4025282763 wait complete. 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.245+0000 7f3501dc1700 1 Processor -- start 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.245+0000 7f3501dc1700 1 -- start start 2024-02-19T18:55:12.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.245+0000 7f3501dc1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc19c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f3501dc1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc19c820 con 0x7f34fc07ad80 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34fb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc19c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34fb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc19c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43800/0 (socket says 172.21.15.39:43800) 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34fb7fe700 1 -- 172.21.15.39:0/2185893541 learned_addr learned my addr 172.21.15.39:0/2185893541 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34fb7fe700 1 -- 172.21.15.39:0/2185893541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec009550 con 0x7f34fc07ad80 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34fb7fe700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc19c2e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f34ec003f80 tx=0x7f34ec011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ec003a80 con 0x7f34fc07ad80 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f34ec003be0 con 0x7f34fc07ad80 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34fc19ca20 con 0x7f34fc07ad80 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.246+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc19cec0 con 0x7f34fc07ad80 2024-02-19T18:55:12.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.247+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ec019820 con 0x7f34fc07ad80 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.247+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f34ec019af0 con 0x7f34fc07ad80 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.248+0000 7f34f8ff9700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.248+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 --> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34e403aef0 con 0x7f34e4038320 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.248+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34ec04d520 con 0x7f34fc07ad80 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.248+0000 7f34faffd700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 msgr2=0x7f34e403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.248+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.449+0000 7f34faffd700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 msgr2=0x7f34e403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.449+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.850+0000 7f34faffd700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 msgr2=0x7f34e403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:06.850+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:07.651+0000 7f34faffd700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 msgr2=0x7f34e403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:12.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:07.651+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:09.252+0000 7f34faffd700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 msgr2=0x7f34e403a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/728525375 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:09.252+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:10.765+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7f34ec02dce0 con 0x7f34fc07ad80 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:10.765+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 msgr2=0x7f34e403a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:10.765+0000 7f34f8ff9700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.770+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f34ec020c50 con 0x7f34fc07ad80 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.770+0000 7f34f8ff9700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f34e403bc00 0x7f34e403dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.770+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34e403aef0 con 0x7f34e403bc00 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.771+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f34e403bc00 0x7f34e403dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.771+0000 7f34faffd700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f34e403bc00 0x7f34e403dff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f34f0003a10 tx=0x7f34f00092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.772+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f34e403aef0 con 0x7f34e403bc00 2024-02-19T18:55:12.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.778+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f34fc19d620 con 0x7f34e403bc00 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.778+0000 7f34f8ff9700 1 -- 172.21.15.39:0/2185893541 <== mgr.14120 v2:172.21.15.39:6800/2039816448 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f34fc19d620 con 0x7f34e403bc00 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f34e403bc00 msgr2=0x7f34e403dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f34e403bc00 0x7f34e403dff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f34f0003a10 tx=0x7f34f00092b0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 msgr2=0x7f34fc19c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc19c2e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f34ec003f80 tx=0x7f34ec011ab0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 shutdown_connections 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f34e403bc00 0x7f34e403dff0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34fc07ad80 0x7f34fc19c2e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 --2- 172.21.15.39:0/2185893541 >> [v2:172.21.15.39:6800/728525375,v1:172.21.15.39:6801/728525375] conn(0x7f34e4038320 0x7f34e403a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 >> 172.21.15.39:0/2185893541 conn(0x7f34fc1019b0 msgr2=0x7f34fc1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 shutdown_connections 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:11.779+0000 7f3501dc1700 1 -- 172.21.15.39:0/2185893541 wait complete. 2024-02-19T18:55:12.451 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 5 is available 2024-02-19T18:55:12.452 INFO:teuthology.orchestra.run.smithi039.stdout:Setting orchestrator backend to cephadm... 2024-02-19T18:55:12.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:12 smithi039 ceph-mon[28988]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T18:55:12.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:12 smithi039 ceph-mon[28988]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T18:55:12.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:12 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:12.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:12 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:12.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:12 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:13.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f4456962700 1 Processor -- start 2024-02-19T18:55:13.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f4456962700 1 -- start start 2024-02-19T18:55:13.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f4456962700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:13.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f4456962700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4450105d20 con 0x7f4450105330 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f444ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f444ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41092/0 (socket says 172.21.15.39:41092) 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f444ffff700 1 -- 172.21.15.39:0/489830439 learned_addr learned my addr 172.21.15.39:0/489830439 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f444ffff700 1 -- 172.21.15.39:0/489830439 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450106530 con 0x7f4450105330 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.825+0000 7f444ffff700 1 --2- 172.21.15.39:0/489830439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450105750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44400098a0 tx=0x7f4440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b524c5442ebbfe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f444effd700 1 -- 172.21.15.39:0/489830439 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4440004180 con 0x7f4450105330 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f444effd700 1 -- 172.21.15.39:0/489830439 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f44400042e0 con 0x7f4450105330 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f444effd700 1 -- 172.21.15.39:0/489830439 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4440011a50 con 0x7f4450105330 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 -- 172.21.15.39:0/489830439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 msgr2=0x7f4450105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 --2- 172.21.15.39:0/489830439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450105750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44400098a0 tx=0x7f4440009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 -- 172.21.15.39:0/489830439 shutdown_connections 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 --2- 172.21.15.39:0/489830439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450105750 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 -- 172.21.15.39:0/489830439 >> 172.21.15.39:0/489830439 conn(0x7f44501008b0 msgr2=0x7f4450102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 -- 172.21.15.39:0/489830439 shutdown_connections 2024-02-19T18:55:13.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.826+0000 7f4456962700 1 -- 172.21.15.39:0/489830439 wait complete. 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f4456962700 1 Processor -- start 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f4456962700 1 -- start start 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f4456962700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450079ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f4456962700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445019ce40 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f444ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450079ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f444ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450079ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41102/0 (socket says 172.21.15.39:41102) 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.827+0000 7f444ffff700 1 -- 172.21.15.39:0/365757120 learned_addr learned my addr 172.21.15.39:0/365757120 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f444ffff700 1 -- 172.21.15.39:0/365757120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4440009550 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f444ffff700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450079ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4440003de0 tx=0x7f4440011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4440003930 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4440003a90 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445019cf80 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44400196c0 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.828+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445019d420 con 0x7f4450105330 2024-02-19T18:55:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.829+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f44400199a0 con 0x7f4450105330 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.829+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44500623c0 con 0x7f4450105330 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.829+0000 7f444d7fa700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f4438038200 0x7f443803a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.829+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f444004c230 con 0x7f4450105330 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.829+0000 7f444f7fe700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f4438038200 0x7f443803a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.829+0000 7f444f7fe700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f4438038200 0x7f443803a6c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4444006fd0 tx=0x7f4444006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:12.834+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4440023370 con 0x7f4450105330 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.008+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f444000fda0 con 0x7f4450105330 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.052+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4450109730 con 0x7f4438038200 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.057+0000 7f444d7fa700 1 -- 172.21.15.39:0/365757120 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4450109730 con 0x7f4438038200 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f4438038200 msgr2=0x7f443803a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f4438038200 0x7f443803a6c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4444006fd0 tx=0x7f4444006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 msgr2=0x7f4450079ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:13.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450079ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4440003de0 tx=0x7f4440011ab0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 shutdown_connections 2024-02-19T18:55:13.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f4438038200 0x7f443803a6c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 --2- 172.21.15.39:0/365757120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4450105330 0x7f4450079ba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 >> 172.21.15.39:0/365757120 conn(0x7f44501008b0 msgr2=0x7f4450101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:13.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 shutdown_connections 2024-02-19T18:55:13.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:13.059+0000 7f4456962700 1 -- 172.21.15.39:0/365757120 wait complete. 2024-02-19T18:55:13.887 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:12] ENGINE Bus STARTING 2024-02-19T18:55:13.887 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:12] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T18:55:13.888 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:12] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T18:55:13.888 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:12] ENGINE Bus STARTED 2024-02-19T18:55:13.888 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: mgrmap e8: smithi039.ydmpuh(active, since 2s) 2024-02-19T18:55:13.888 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:13.888 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:13 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.077+0000 7f0da9618700 1 Processor -- start 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.077+0000 7f0da9618700 1 -- start start 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.077+0000 7f0da9618700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da4104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.077+0000 7f0da9618700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da41054b0 con 0x7f0da4104ac0 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da4104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da4104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41106/0 (socket says 172.21.15.39:41106) 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da2ffd700 1 -- 172.21.15.39:0/2630135158 learned_addr learned my addr 172.21.15.39:0/2630135158 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:14.831 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da2ffd700 1 -- 172.21.15.39:0/2630135158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4105cc0 con 0x7f0da4104ac0 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da2ffd700 1 --2- 172.21.15.39:0/2630135158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da4104ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00aa40 tx=0x7f0d9c00ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeebfd6bd33ca5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da1ffb700 1 -- 172.21.15.39:0/2630135158 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c004180 con 0x7f0da4104ac0 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da1ffb700 1 -- 172.21.15.39:0/2630135158 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0d9c00cd60 con 0x7f0da4104ac0 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.078+0000 7f0da1ffb700 1 -- 172.21.15.39:0/2630135158 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c003ce0 con 0x7f0da4104ac0 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 -- 172.21.15.39:0/2630135158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 msgr2=0x7f0da4104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 --2- 172.21.15.39:0/2630135158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da4104ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00aa40 tx=0x7f0d9c00ad50 comp rx=0 tx=0).stop 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 -- 172.21.15.39:0/2630135158 shutdown_connections 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 --2- 172.21.15.39:0/2630135158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da4104ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 -- 172.21.15.39:0/2630135158 >> 172.21.15.39:0/2630135158 conn(0x7f0da4100040 msgr2=0x7f0da41024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 -- 172.21.15.39:0/2630135158 shutdown_connections 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 -- 172.21.15.39:0/2630135158 wait complete. 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.079+0000 7f0da9618700 1 Processor -- start 2024-02-19T18:55:14.832 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da9618700 1 -- start start 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da9618700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da9618700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da419a680 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41122/0 (socket says 172.21.15.39:41122) 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da2ffd700 1 -- 172.21.15.39:0/880061910 learned_addr learned my addr 172.21.15.39:0/880061910 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da2ffd700 1 -- 172.21.15.39:0/880061910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c00a6f0 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0da2ffd700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da419a140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0041d0 tx=0x7f0d9c0042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.080+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c00ced0 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.081+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0d9c010a70 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.081+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c021870 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.081+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da419a880 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.081+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da419ad20 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.081+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0d9c02b440 con 0x7f0da4104ac0 2024-02-19T18:55:14.833 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.081+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da4193a20 con 0x7f0da4104ac0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.082+0000 7f0d9bfff700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0d900382d0 0x7f0d9003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.082+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d9c04bfd0 con 0x7f0da4104ac0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.082+0000 7f0da27fc700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0d900382d0 0x7f0d9003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.082+0000 7f0da27fc700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0d900382d0 0x7f0d9003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c006fd0 tx=0x7f0d8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.087+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0d9c07f0e0 con 0x7f0da4104ac0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.285+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0da40611d0 con 0x7f0d900382d0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.286+0000 7f0d9bfff700 1 -- 172.21.15.39:0/880061910 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0da40611d0 con 0x7f0d900382d0 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0d900382d0 msgr2=0x7f0d9003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0d900382d0 0x7f0d9003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c006fd0 tx=0x7f0d8c006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 msgr2=0x7f0da419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da419a140 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0041d0 tx=0x7f0d9c0042b0 comp rx=0 tx=0).stop 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 shutdown_connections 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0d900382d0 0x7f0d9003a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 --2- 172.21.15.39:0/880061910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0da4104ac0 0x7f0da419a140 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 >> 172.21.15.39:0/880061910 conn(0x7f0da4100040 msgr2=0x7f0da418e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:14.835 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 shutdown_connections 2024-02-19T18:55:14.835 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:14.288+0000 7f0da9618700 1 -- 172.21.15.39:0/880061910 wait complete. 2024-02-19T18:55:14.835 INFO:teuthology.orchestra.run.smithi039.stdout:Generating ssh key... 2024-02-19T18:55:15.218 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:14 smithi039 ceph-mon[28988]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:16.131 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:15 smithi039 ceph-mon[28988]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:16.131 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:15 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:16.132 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:15 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:16.139 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.215+0000 7f5ac586e700 1 Processor -- start 2024-02-19T18:55:16.139 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5ac586e700 1 -- start start 2024-02-19T18:55:16.139 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5ac586e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac0106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:16.139 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5ac586e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac0106e10 con 0x7f5ac0106420 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41124/0 (socket says 172.21.15.39:41124) 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abeffd700 1 -- 172.21.15.39:0/1224995478 learned_addr learned my addr 172.21.15.39:0/1224995478 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abeffd700 1 -- 172.21.15.39:0/1224995478 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac0107620 con 0x7f5ac0106420 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abeffd700 1 --2- 172.21.15.39:0/1224995478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac0106840 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80098a0 tx=0x7f5aa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e3eeb2150ae04b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abdffb700 1 -- 172.21.15.39:0/1224995478 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa8004180 con 0x7f5ac0106420 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abdffb700 1 -- 172.21.15.39:0/1224995478 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5aa80042e0 con 0x7f5ac0106420 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.216+0000 7f5abdffb700 1 -- 172.21.15.39:0/1224995478 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa8011a50 con 0x7f5ac0106420 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 -- 172.21.15.39:0/1224995478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 msgr2=0x7f5ac0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:16.140 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 --2- 172.21.15.39:0/1224995478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac0106840 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80098a0 tx=0x7f5aa8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 -- 172.21.15.39:0/1224995478 shutdown_connections 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 --2- 172.21.15.39:0/1224995478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac0106840 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 -- 172.21.15.39:0/1224995478 >> 172.21.15.39:0/1224995478 conn(0x7f5ac0101a00 msgr2=0x7f5ac0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 -- 172.21.15.39:0/1224995478 shutdown_connections 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.217+0000 7f5ac586e700 1 -- 172.21.15.39:0/1224995478 wait complete. 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5ac586e700 1 Processor -- start 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5ac586e700 1 -- start start 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5ac586e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5ac586e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019a5f0 con 0x7f5ac0106420 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5abeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:16.141 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5abeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41136/0 (socket says 172.21.15.39:41136) 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5abeffd700 1 -- 172.21.15.39:0/3333778273 learned_addr learned my addr 172.21.15.39:0/3333778273 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5abeffd700 1 -- 172.21.15.39:0/3333778273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8009550 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.218+0000 7f5abeffd700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac019a0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8006af0 tx=0x7f5aa8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.219+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa8003b90 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.219+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5aa801a8a0 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.219+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac019a7f0 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.219+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa8019880 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.219+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac019ac90 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.220+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5aa801a430 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.220+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac00623c0 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.220+0000 7f5ac486c700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f5aac0382e0 0x7f5aac03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.220+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5aa804c6d0 con 0x7f5ac0106420 2024-02-19T18:55:16.142 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.220+0000 7f5abe7fc700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f5aac0382e0 0x7f5aac03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.220+0000 7f5abe7fc700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f5aac0382e0 0x7f5aac03a7a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0006fd0 tx=0x7f5ab0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.225+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5aa8018b80 con 0x7f5ac0106420 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.427+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ac010a820 con 0x7f5aac0382e0 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.519+0000 7f5ac486c700 1 -- 172.21.15.39:0/3333778273 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5ac010a820 con 0x7f5aac0382e0 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f5aac0382e0 msgr2=0x7f5aac03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f5aac0382e0 0x7f5aac03a7a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0006fd0 tx=0x7f5ab0006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 msgr2=0x7f5ac019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac019a0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8006af0 tx=0x7f5aa8011a90 comp rx=0 tx=0).stop 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 shutdown_connections 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f5aac0382e0 0x7f5aac03a7a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 --2- 172.21.15.39:0/3333778273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ac0106420 0x7f5ac019a0b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:16.143 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 >> 172.21.15.39:0/3333778273 conn(0x7f5ac0101a00 msgr2=0x7f5ac01027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:16.144 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 shutdown_connections 2024-02-19T18:55:16.144 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:15.521+0000 7f5ac586e700 1 -- 172.21.15.39:0/3333778273 wait complete. 2024-02-19T18:55:17.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:16 smithi039 ceph-mon[28988]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:17.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:16 smithi039 ceph-mon[28988]: Generating ssh key... 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm9U5NY9dpvY/KtGw3Vy0jmjxpe5V8EsBCPIdrBqCd8MkySdOXdXjEXFCIadsv/W/zY2QBtHxEQGL6Xi+1atEb2DhWr6yO5vh7MaM6kk4idHQm/wyXcmv0IyXSVhjv7XfcDmaXvNOSYgnToqjJ+KQqj1PlajydYwsfldl0N48r6hEgP0IGw4RU7SfZ6W+RAHL3haXiFUywSBi36HcthpImnpBrRu6P04QoKQmtnOJbqDtOjwostOty42mL1VdBiFcfJEEsIrLJzaYrgoIy7nf+a0uCVqo/Se3CLx2xkdSrscYYg7F8OTnrPxkSMHz578D/9nOrgXteZ0wkV9yrs2Kz5U7PIxVrHhRQ8sZx+zp1W5BVAQ26g9DVNKrbUrVETJYSyNkzGfEUngdyyWim+SPjOR3cfzxZWOQo3WIxfzu62ASHvs35t6gjINy+8EnSsUvE4fAMntxpyYAn67O+RED2yObPtZMDavOMlWvynJnVdd7ossx88KZldQ0CyeBl2s8= ceph-1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.544+0000 7fb15159e700 1 Processor -- start 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.544+0000 7fb15159e700 1 -- start start 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.544+0000 7fb15159e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c109330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.544+0000 7fb15159e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c079e40 con 0x7fb14c106f40 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54134/0 (socket says 172.21.15.39:54134) 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14affd700 1 -- 172.21.15.39:0/497156839 learned_addr learned my addr 172.21.15.39:0/497156839 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14affd700 1 -- 172.21.15.39:0/497156839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c109870 con 0x7fb14c106f40 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14affd700 1 --2- 172.21.15.39:0/497156839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c109330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0098a0 tx=0x7fb13c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9502149a232726e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:17.296 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14a7fc700 1 -- 172.21.15.39:0/497156839 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c004180 con 0x7fb14c106f40 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14a7fc700 1 -- 172.21.15.39:0/497156839 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb13c0042e0 con 0x7fb14c106f40 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.545+0000 7fb14a7fc700 1 -- 172.21.15.39:0/497156839 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c011a50 con 0x7fb14c106f40 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 -- 172.21.15.39:0/497156839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 msgr2=0x7fb14c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 --2- 172.21.15.39:0/497156839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c109330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0098a0 tx=0x7fb13c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 -- 172.21.15.39:0/497156839 shutdown_connections 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 --2- 172.21.15.39:0/497156839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c109330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 -- 172.21.15.39:0/497156839 >> 172.21.15.39:0/497156839 conn(0x7fb14c1008d0 msgr2=0x7fb14c102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 -- 172.21.15.39:0/497156839 shutdown_connections 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.546+0000 7fb15159e700 1 -- 172.21.15.39:0/497156839 wait complete. 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb15159e700 1 Processor -- start 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb15159e700 1 -- start start 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb15159e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb15159e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c19a600 con 0x7fb14c106f40 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb14affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:17.297 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb14affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54144/0 (socket says 172.21.15.39:54144) 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.547+0000 7fb14affd700 1 -- 172.21.15.39:0/1607654586 learned_addr learned my addr 172.21.15.39:0/1607654586 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb14affd700 1 -- 172.21.15.39:0/1607654586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c009550 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb14affd700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c19a0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb13c003f80 tx=0x7fb13c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c003a80 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fb13c003be0 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c19a800 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c019820 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.548+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c19aca0 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.549+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb13c019b20 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.549+0000 7fb148ff9700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fb1380382d0 0x7fb13803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.549+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14c10a590 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.549+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb13c04c5a0 con 0x7fb14c106f40 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.549+0000 7fb143fff700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fb1380382d0 0x7fb13803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:17.298 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.550+0000 7fb143fff700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fb1380382d0 0x7fb13803a790 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb134006fd0 tx=0x7fb134006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.554+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb13c02c3a0 con 0x7fb14c106f40 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.753+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb14c101da0 con 0x7fb1380382d0 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.754+0000 7fb148ff9700 1 -- 172.21.15.39:0/1607654586 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb14c101da0 con 0x7fb1380382d0 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fb1380382d0 msgr2=0x7fb13803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fb1380382d0 0x7fb13803a790 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb134006fd0 tx=0x7fb134006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 msgr2=0x7fb14c19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c19a0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb13c003f80 tx=0x7fb13c011ab0 comp rx=0 tx=0).stop 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 shutdown_connections 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fb1380382d0 0x7fb13803a790 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 --2- 172.21.15.39:0/1607654586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb14c106f40 0x7fb14c19a0c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 >> 172.21.15.39:0/1607654586 conn(0x7fb14c1008d0 msgr2=0x7fb14c101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 shutdown_connections 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:16.756+0000 7fb15159e700 1 -- 172.21.15.39:0/1607654586 wait complete. 2024-02-19T18:55:17.299 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T18:55:17.300 INFO:teuthology.orchestra.run.smithi039.stdout:Adding key to root@localhost authorized_keys... 2024-02-19T18:55:17.300 INFO:teuthology.orchestra.run.smithi039.stdout:Adding host smithi039... 2024-02-19T18:55:18.148 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:17 smithi039 ceph-mon[28988]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:19.080 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:18 smithi039 ceph-mon[28988]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:20.166 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:19 smithi039 ceph-mon[28988]: Deploying cephadm binary to smithi039 2024-02-19T18:55:21.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Added host 'smithi039' with addr '172.21.15.39' 2024-02-19T18:55:21.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9348049700 1 Processor -- start 2024-02-19T18:55:21.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9348049700 1 -- start start 2024-02-19T18:55:21.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9348049700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:21.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9348049700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9340079ec0 con 0x7f9340106ea0 2024-02-19T18:55:21.181 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9345de5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9345de5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54156/0 (socket says 172.21.15.39:54156) 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9345de5700 1 -- 172.21.15.39:0/2993870526 learned_addr learned my addr 172.21.15.39:0/2993870526 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.669+0000 7f9345de5700 1 -- 172.21.15.39:0/2993870526 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93401097d0 con 0x7f9340106ea0 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9345de5700 1 --2- 172.21.15.39:0/2993870526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340109290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f933c0098a0 tx=0x7f933c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19e2ef029c21af0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9344de3700 1 -- 172.21.15.39:0/2993870526 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f933c004180 con 0x7f9340106ea0 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9344de3700 1 -- 172.21.15.39:0/2993870526 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f933c0042e0 con 0x7f9340106ea0 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9344de3700 1 -- 172.21.15.39:0/2993870526 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f933c011a50 con 0x7f9340106ea0 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9348049700 1 -- 172.21.15.39:0/2993870526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 msgr2=0x7f9340109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9348049700 1 --2- 172.21.15.39:0/2993870526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340109290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f933c0098a0 tx=0x7f933c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9348049700 1 -- 172.21.15.39:0/2993870526 shutdown_connections 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9348049700 1 --2- 172.21.15.39:0/2993870526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340109290 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:21.182 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.670+0000 7f9348049700 1 -- 172.21.15.39:0/2993870526 >> 172.21.15.39:0/2993870526 conn(0x7f93401008b0 msgr2=0x7f9340102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9348049700 1 -- 172.21.15.39:0/2993870526 shutdown_connections 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9348049700 1 -- 172.21.15.39:0/2993870526 wait complete. 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9348049700 1 Processor -- start 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9348049700 1 -- start start 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9348049700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9348049700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9340106d40 con 0x7f9340106ea0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.671+0000 7f9345de5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9345de5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54172/0 (socket says 172.21.15.39:54172) 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9345de5700 1 -- 172.21.15.39:0/2307178628 learned_addr learned my addr 172.21.15.39:0/2307178628 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9345de5700 1 -- 172.21.15.39:0/2307178628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c009550 con 0x7f9340106ea0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9345de5700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340106800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f933c003f80 tx=0x7f933c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f933c003a80 con 0x7f9340106ea0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f933c003be0 con 0x7f9340106ea0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f933c019820 con 0x7f9340106ea0 2024-02-19T18:55:21.183 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9340104e50 con 0x7f9340106ea0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.672+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93401052f0 con 0x7f9340106ea0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.673+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f933c019a50 con 0x7f9340106ea0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.673+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9340062430 con 0x7f9340106ea0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.673+0000 7f9336ffd700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f932c038320 0x7f932c03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.673+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f933c04c5e0 con 0x7f9340106ea0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.673+0000 7f93455e4700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f932c038320 0x7f932c03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.674+0000 7f93455e4700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f932c038320 0x7f932c03a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9330006fd0 tx=0x7f9330006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.678+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9340062430 con 0x7f9340106ea0 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:17.879+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}) v1 -- 0x7f93400623c0 con 0x7f932c038320 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.527+0000 7f9336ffd700 1 -- 172.21.15.39:0/2307178628 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f93400623c0 con 0x7f932c038320 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f932c038320 msgr2=0x7f932c03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f932c038320 0x7f932c03a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9330006fd0 tx=0x7f9330006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:21.184 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 msgr2=0x7f9340106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340106800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f933c003f80 tx=0x7f933c011ab0 comp rx=0 tx=0).stop 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 shutdown_connections 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f932c038320 0x7f932c03a7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 --2- 172.21.15.39:0/2307178628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9340106ea0 0x7f9340106800 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 >> 172.21.15.39:0/2307178628 conn(0x7f93401008b0 msgr2=0x7f9340101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 shutdown_connections 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:20.530+0000 7f9348049700 1 -- 172.21.15.39:0/2307178628 wait complete. 2024-02-19T18:55:21.185 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying mon service with default placement... 2024-02-19T18:55:21.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:21 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:21.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:21 smithi039 ceph-mon[28988]: Added host smithi039 2024-02-19T18:55:21.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:21 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:22.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-19T18:55:22.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.541+0000 7ff4b3a92700 1 Processor -- start 2024-02-19T18:55:22.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b3a92700 1 -- start start 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b3a92700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac07a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b3a92700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac07a8e0 con 0x7ff4ac07beb0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b182e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac07a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b182e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac07a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54186/0 (socket says 172.21.15.39:54186) 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b182e700 1 -- 172.21.15.39:0/238316328 learned_addr learned my addr 172.21.15.39:0/238316328 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b182e700 1 -- 172.21.15.39:0/238316328 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac07aa20 con 0x7ff4ac07beb0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.542+0000 7ff4b182e700 1 --2- 172.21.15.39:0/238316328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac07a310 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009e80 tx=0x7ff49c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631fbd1ae4f2e714 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b082c700 1 -- 172.21.15.39:0/238316328 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49c00e070 con 0x7ff4ac07beb0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b082c700 1 -- 172.21.15.39:0/238316328 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff49c003e90 con 0x7ff4ac07beb0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b082c700 1 -- 172.21.15.39:0/238316328 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49c004380 con 0x7ff4ac07beb0 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 -- 172.21.15.39:0/238316328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 msgr2=0x7ff4ac07a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 --2- 172.21.15.39:0/238316328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac07a310 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009e80 tx=0x7ff49c00b330 comp rx=0 tx=0).stop 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 -- 172.21.15.39:0/238316328 shutdown_connections 2024-02-19T18:55:22.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 --2- 172.21.15.39:0/238316328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac07a310 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 -- 172.21.15.39:0/238316328 >> 172.21.15.39:0/238316328 conn(0x7ff4ac101a00 msgr2=0x7ff4ac103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 -- 172.21.15.39:0/238316328 shutdown_connections 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.543+0000 7ff4b3a92700 1 -- 172.21.15.39:0/238316328 wait complete. 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b3a92700 1 Processor -- start 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b3a92700 1 -- start start 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b3a92700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b3a92700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19c7d0 con 0x7ff4ac07beb0 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b182e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b182e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac19c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54202/0 (socket says 172.21.15.39:54202) 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b182e700 1 -- 172.21.15.39:0/4251820128 learned_addr learned my addr 172.21.15.39:0/4251820128 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.544+0000 7ff4b182e700 1 -- 172.21.15.39:0/4251820128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c009b30 con 0x7ff4ac07beb0 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.545+0000 7ff4b182e700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac19c290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009530 tx=0x7ff49c004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.545+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49c00e040 con 0x7ff4ac07beb0 2024-02-19T18:55:22.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.545+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff49c003c30 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.545+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac19c9d0 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.545+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49c01b740 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.545+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac19ced0 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.546+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff49c019040 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.546+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac10c780 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.546+0000 7ff4aaffd700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff494038370 0x7ff49403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.546+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff49c04bc70 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.546+0000 7ff4b102d700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff494038370 0x7ff49403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.546+0000 7ff4b102d700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff494038370 0x7ff49403a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0006fd0 tx=0x7ff4a0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.551+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff49c011980 con 0x7ff4ac07beb0 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.770+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4ac102eb0 con 0x7ff494038370 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.775+0000 7ff4aaffd700 1 -- 172.21.15.39:0/4251820128 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff4ac102eb0 con 0x7ff494038370 2024-02-19T18:55:22.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff494038370 msgr2=0x7ff49403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff494038370 0x7ff49403a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0006fd0 tx=0x7ff4a0006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 msgr2=0x7ff4ac19c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac19c290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009530 tx=0x7ff49c004920 comp rx=0 tx=0).stop 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 shutdown_connections 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff494038370 0x7ff49403a830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 --2- 172.21.15.39:0/4251820128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4ac07beb0 0x7ff4ac19c290 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 >> 172.21.15.39:0/4251820128 conn(0x7ff4ac101a00 msgr2=0x7ff4ac1027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 shutdown_connections 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:21.777+0000 7ff4b3a92700 1 -- 172.21.15.39:0/4251820128 wait complete. 2024-02-19T18:55:22.460 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying mgr service with default placement... 2024-02-19T18:55:23.075 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:22 smithi039 ceph-mon[28988]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:23.075 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:22 smithi039 ceph-mon[28988]: Saving service mon spec with placement count:5 2024-02-19T18:55:23.075 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:22 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:24.344 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-19T18:55:24.344 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36cd08700 1 Processor -- start 2024-02-19T18:55:24.344 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36cd08700 1 -- start start 2024-02-19T18:55:24.344 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36cd08700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff368105110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:24.344 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36cd08700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368079e60 con 0x7ff368104cf0 2024-02-19T18:55:24.344 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff368105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff368105110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54218/0 (socket says 172.21.15.39:54218) 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36659c700 1 -- 172.21.15.39:0/1968566330 learned_addr learned my addr 172.21.15.39:0/1968566330 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.229+0000 7ff36659c700 1 -- 172.21.15.39:0/1968566330 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368105650 con 0x7ff368104cf0 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36659c700 1 --2- 172.21.15.39:0/1968566330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff368105110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff358009e80 tx=0x7ff35800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef097765f0d97cd0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36559a700 1 -- 172.21.15.39:0/1968566330 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35800e070 con 0x7ff368104cf0 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36559a700 1 -- 172.21.15.39:0/1968566330 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff358003e90 con 0x7ff368104cf0 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36559a700 1 -- 172.21.15.39:0/1968566330 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff358004380 con 0x7ff368104cf0 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 -- 172.21.15.39:0/1968566330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 msgr2=0x7ff368105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:24.345 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 --2- 172.21.15.39:0/1968566330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff368105110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff358009e80 tx=0x7ff35800b330 comp rx=0 tx=0).stop 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 -- 172.21.15.39:0/1968566330 shutdown_connections 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 --2- 172.21.15.39:0/1968566330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff368105110 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 -- 172.21.15.39:0/1968566330 >> 172.21.15.39:0/1968566330 conn(0x7ff3681008b0 msgr2=0x7ff368102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 -- 172.21.15.39:0/1968566330 shutdown_connections 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.230+0000 7ff36cd08700 1 -- 172.21.15.39:0/1968566330 wait complete. 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36cd08700 1 Processor -- start 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36cd08700 1 -- start start 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36cd08700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff36819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36cd08700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36819c800 con 0x7ff368104cf0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff36819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff36819c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54222/0 (socket says 172.21.15.39:54222) 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.231+0000 7ff36659c700 1 -- 172.21.15.39:0/3667610606 learned_addr learned my addr 172.21.15.39:0/3667610606 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff36659c700 1 -- 172.21.15.39:0/3667610606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358009b30 con 0x7ff368104cf0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff36659c700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff36819c2c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff35800d010 tx=0x7ff358004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35800e040 con 0x7ff368104cf0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36819ca00 con 0x7ff368104cf0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff358003c30 con 0x7ff368104cf0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff35801b790 con 0x7ff368104cf0 2024-02-19T18:55:24.347 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.232+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36819cf00 con 0x7ff368104cf0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.233+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff358019040 con 0x7ff368104cf0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.233+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff368195ba0 con 0x7ff368104cf0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.233+0000 7ff35f7fe700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff3540382e0 0x7ff35403a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.233+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff35804bfe0 con 0x7ff368104cf0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.233+0000 7ff365d9b700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff3540382e0 0x7ff35403a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.234+0000 7ff365d9b700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff3540382e0 0x7ff35403a7a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff350006fd0 tx=0x7ff350006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.238+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff358011390 con 0x7ff368104cf0 2024-02-19T18:55:24.348 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.449+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3680611d0 con 0x7ff3540382e0 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.458+0000 7ff35f7fe700 1 -- 172.21.15.39:0/3667610606 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff3680611d0 con 0x7ff3540382e0 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff3540382e0 msgr2=0x7ff35403a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff3540382e0 0x7ff35403a7a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff350006fd0 tx=0x7ff350006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 msgr2=0x7ff36819c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff36819c2c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff35800d010 tx=0x7ff358004430 comp rx=0 tx=0).stop 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 shutdown_connections 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ff3540382e0 0x7ff35403a7a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:24.349 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 --2- 172.21.15.39:0/3667610606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff368104cf0 0x7ff36819c2c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:24.350 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 >> 172.21.15.39:0/3667610606 conn(0x7ff3681008b0 msgr2=0x7ff3681908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:24.350 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 shutdown_connections 2024-02-19T18:55:24.350 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:23.461+0000 7ff36cd08700 1 -- 172.21.15.39:0/3667610606 wait complete. 2024-02-19T18:55:24.350 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying crash service with default placement... 2024-02-19T18:55:24.638 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:24 smithi039 ceph-mon[28988]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:24.638 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:24 smithi039 ceph-mon[28988]: Saving service mgr spec with placement count:2 2024-02-19T18:55:24.638 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:24 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:24.639 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:24 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:25.585 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-19T18:55:25.585 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.723+0000 7f05a6c88700 1 Processor -- start 2024-02-19T18:55:25.585 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.723+0000 7f05a6c88700 1 -- start start 2024-02-19T18:55:25.585 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.723+0000 7f05a6c88700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0106d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.723+0000 7f05a6c88700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a01072e0 con 0x7f05a01068f0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.723+0000 7f05a4a24700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0106d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f05a4a24700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0106d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54236/0 (socket says 172.21.15.39:54236) 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f05a4a24700 1 -- 172.21.15.39:0/3003444651 learned_addr learned my addr 172.21.15.39:0/3003444651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f05a4a24700 1 -- 172.21.15.39:0/3003444651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0105550 con 0x7f05a01068f0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f05a4a24700 1 --2- 172.21.15.39:0/3003444651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0106d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f059400bb40 tx=0x7f059400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97756e94dfbb97e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f059f7fe700 1 -- 172.21.15.39:0/3003444651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059400dc00 con 0x7f05a01068f0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f059f7fe700 1 -- 172.21.15.39:0/3003444651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f05940044e0 con 0x7f05a01068f0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.724+0000 7f059f7fe700 1 -- 172.21.15.39:0/3003444651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594010470 con 0x7f05a01068f0 2024-02-19T18:55:25.586 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 -- 172.21.15.39:0/3003444651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 msgr2=0x7f05a0106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 --2- 172.21.15.39:0/3003444651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0106d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f059400bb40 tx=0x7f059400be50 comp rx=0 tx=0).stop 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 -- 172.21.15.39:0/3003444651 shutdown_connections 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 --2- 172.21.15.39:0/3003444651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0106d10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 -- 172.21.15.39:0/3003444651 >> 172.21.15.39:0/3003444651 conn(0x7f05a0100040 msgr2=0x7f05a01024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 -- 172.21.15.39:0/3003444651 shutdown_connections 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 -- 172.21.15.39:0/3003444651 wait complete. 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 Processor -- start 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.725+0000 7f05a6c88700 1 -- start start 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a6c88700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a6c88700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a0198490 con 0x7f05a01068f0 2024-02-19T18:55:25.587 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a4a24700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a4a24700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54242/0 (socket says 172.21.15.39:54242) 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a4a24700 1 -- 172.21.15.39:0/314692819 learned_addr learned my addr 172.21.15.39:0/314692819 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a4a24700 1 -- 172.21.15.39:0/314692819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059400b7f0 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a4a24700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0197f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0594003ba0 tx=0x7f0594004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f059401e070 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a0198690 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f059400de20 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.726+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0594022940 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.727+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a0198b30 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.727+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f059402c440 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.728+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a0191bf0 con 0x7f05a01068f0 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.728+0000 7f059dffb700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0590038320 0x7f059003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.728+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f059404e630 con 0x7f05a01068f0 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.728+0000 7f059ffff700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0590038320 0x7f059003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.728+0000 7f059ffff700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0590038320 0x7f059003a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f058c006fd0 tx=0x7f058c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.736+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f059402a950 con 0x7f05a01068f0 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.951+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f05a00611d0 con 0x7f0590038320 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.958+0000 7f059dffb700 1 -- 172.21.15.39:0/314692819 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f05a00611d0 con 0x7f0590038320 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0590038320 msgr2=0x7f059003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0590038320 0x7f059003a7e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f058c006fd0 tx=0x7f058c006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 msgr2=0x7f05a0197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0197f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0594003ba0 tx=0x7f0594004070 comp rx=0 tx=0).stop 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 shutdown_connections 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0590038320 0x7f059003a7e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 --2- 172.21.15.39:0/314692819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f05a01068f0 0x7f05a0197f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 >> 172.21.15.39:0/314692819 conn(0x7f05a0100040 msgr2=0x7f05a018ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 shutdown_connections 2024-02-19T18:55:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:24.960+0000 7f05a6c88700 1 -- 172.21.15.39:0/314692819 wait complete. 2024-02-19T18:55:25.590 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying ceph-exporter service with default placement... 2024-02-19T18:55:25.920 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:25 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:25.921 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:25 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:26.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:26 smithi039 ceph-mon[28988]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:26.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:26 smithi039 ceph-mon[28988]: Saving service crash spec with placement * 2024-02-19T18:55:26.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:26 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:27.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-02-19T18:55:27.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.003+0000 7fc71f0c0700 1 Processor -- start 2024-02-19T18:55:27.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.003+0000 7fc71f0c0700 1 -- start start 2024-02-19T18:55:27.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71f0c0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:27.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71f0c0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc718105d20 con 0x7fc718105330 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71ce5c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71ce5c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54244/0 (socket says 172.21.15.39:54244) 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71ce5c700 1 -- 172.21.15.39:0/683099791 learned_addr learned my addr 172.21.15.39:0/683099791 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71ce5c700 1 -- 172.21.15.39:0/683099791 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc718106530 con 0x7fc718105330 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc71ce5c700 1 --2- 172.21.15.39:0/683099791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718105750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0098a0 tx=0x7fc70c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b07e8e19c75b72e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc7177fe700 1 -- 172.21.15.39:0/683099791 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc70c004180 con 0x7fc718105330 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc7177fe700 1 -- 172.21.15.39:0/683099791 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc70c0042e0 con 0x7fc718105330 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.004+0000 7fc7177fe700 1 -- 172.21.15.39:0/683099791 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc70c011af0 con 0x7fc718105330 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 -- 172.21.15.39:0/683099791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 msgr2=0x7fc718105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 --2- 172.21.15.39:0/683099791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718105750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0098a0 tx=0x7fc70c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 -- 172.21.15.39:0/683099791 shutdown_connections 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 --2- 172.21.15.39:0/683099791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718105750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 -- 172.21.15.39:0/683099791 >> 172.21.15.39:0/683099791 conn(0x7fc7181008b0 msgr2=0x7fc718102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:27.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 -- 172.21.15.39:0/683099791 shutdown_connections 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 -- 172.21.15.39:0/683099791 wait complete. 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.005+0000 7fc71f0c0700 1 Processor -- start 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71f0c0700 1 -- start start 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71f0c0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718190520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71f0c0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc718190a60 con 0x7fc718105330 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71ce5c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718190520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71ce5c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718190520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54246/0 (socket says 172.21.15.39:54246) 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71ce5c700 1 -- 172.21.15.39:0/244516017 learned_addr learned my addr 172.21.15.39:0/244516017 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71ce5c700 1 -- 172.21.15.39:0/244516017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c009550 con 0x7fc718105330 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71ce5c700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718190520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc70c011670 tx=0x7fc70c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc70c0039c0 con 0x7fc718105330 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc70c003b20 con 0x7fc718105330 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71818eb70 con 0x7fc718105330 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.006+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc70c0247f0 con 0x7fc718105330 2024-02-19T18:55:27.329 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.007+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71818f010 con 0x7fc718105330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.007+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fc70c024a60 con 0x7fc718105330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.007+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7180623c0 con 0x7fc718105330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.007+0000 7fc715ffb700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fc708038330 0x7fc70803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.008+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc70c04c410 con 0x7fc718105330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.008+0000 7fc717fff700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fc708038330 0x7fc70803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.008+0000 7fc717fff700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fc708038330 0x7fc70803a7f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc704006fd0 tx=0x7fc704006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.012+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc70c024d10 con 0x7fc718105330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.232+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fc718102b10 con 0x7fc708038330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.239+0000 7fc715ffb700 1 -- 172.21.15.39:0/244516017 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fc718102b10 con 0x7fc708038330 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fc708038330 msgr2=0x7fc70803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fc708038330 0x7fc70803a7f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc704006fd0 tx=0x7fc704006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 msgr2=0x7fc718190520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718190520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc70c011670 tx=0x7fc70c011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:27.330 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 shutdown_connections 2024-02-19T18:55:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fc708038330 0x7fc70803a7f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 --2- 172.21.15.39:0/244516017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc718105330 0x7fc718190520 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 >> 172.21.15.39:0/244516017 conn(0x7fc7181008b0 msgr2=0x7fc7180798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 shutdown_connections 2024-02-19T18:55:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:26.241+0000 7fc71f0c0700 1 -- 172.21.15.39:0/244516017 wait complete. 2024-02-19T18:55:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying prometheus service with default placement... 2024-02-19T18:55:27.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:27 smithi039 ceph-mon[28988]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:27.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:27 smithi039 ceph-mon[28988]: Saving service ceph-exporter spec with placement * 2024-02-19T18:55:27.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:27 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:28.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-19T18:55:28.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.768+0000 7fa55f78f700 1 Processor -- start 2024-02-19T18:55:28.643 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.768+0000 7fa55f78f700 1 -- start start 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.768+0000 7fa55f78f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa5581092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.768+0000 7fa55f78f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558079e80 con 0x7fa558106f00 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa55d52b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa5581092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa55d52b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa5581092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41908/0 (socket says 172.21.15.39:41908) 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa55d52b700 1 -- 172.21.15.39:0/3571254765 learned_addr learned my addr 172.21.15.39:0/3571254765 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa55d52b700 1 -- 172.21.15.39:0/3571254765 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa558109830 con 0x7fa558106f00 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa55d52b700 1 --2- 172.21.15.39:0/3571254765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa5581092f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5480098a0 tx=0x7fa548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d74d403ac45f38 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa54ffff700 1 -- 172.21.15.39:0/3571254765 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa548004180 con 0x7fa558106f00 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa54ffff700 1 -- 172.21.15.39:0/3571254765 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa5480042e0 con 0x7fa558106f00 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.769+0000 7fa54ffff700 1 -- 172.21.15.39:0/3571254765 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa548011af0 con 0x7fa558106f00 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 -- 172.21.15.39:0/3571254765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 msgr2=0x7fa5581092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 --2- 172.21.15.39:0/3571254765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa5581092f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5480098a0 tx=0x7fa548009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.644 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 -- 172.21.15.39:0/3571254765 shutdown_connections 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 --2- 172.21.15.39:0/3571254765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa5581092f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 -- 172.21.15.39:0/3571254765 >> 172.21.15.39:0/3571254765 conn(0x7fa5581008d0 msgr2=0x7fa558102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 -- 172.21.15.39:0/3571254765 shutdown_connections 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.770+0000 7fa55f78f700 1 -- 172.21.15.39:0/3571254765 wait complete. 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55f78f700 1 Processor -- start 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55f78f700 1 -- start start 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55f78f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa55819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55f78f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55819a650 con 0x7fa558106f00 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55d52b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa55819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55d52b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa55819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41914/0 (socket says 172.21.15.39:41914) 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55d52b700 1 -- 172.21.15.39:0/2703010865 learned_addr learned my addr 172.21.15.39:0/2703010865 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:28.645 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55d52b700 1 -- 172.21.15.39:0/2703010865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa548009550 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.771+0000 7fa55d52b700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa55819a110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa548011670 tx=0x7fa548011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.772+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa548003a40 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.772+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55819a850 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.772+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa548003ba0 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.772+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa548024820 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.772+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55819acf0 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.772+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa548024a90 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.773+0000 7fa54e7fc700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fa544038330 0x7fa54403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.773+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa54804c2d0 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.773+0000 7fa55cd2a700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fa544038330 0x7fa54403a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.773+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5580623c0 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.773+0000 7fa55cd2a700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fa544038330 0x7fa54403a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa554006fd0 tx=0x7fa554006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:27.782+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa548022310 con 0x7fa558106f00 2024-02-19T18:55:28.646 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.015+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa558101da0 con 0x7fa544038330 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.022+0000 7fa54e7fc700 1 -- 172.21.15.39:0/2703010865 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa558101da0 con 0x7fa544038330 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fa544038330 msgr2=0x7fa54403a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fa544038330 0x7fa54403a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa554006fd0 tx=0x7fa554006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 msgr2=0x7fa55819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa55819a110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa548011670 tx=0x7fa548011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 shutdown_connections 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fa544038330 0x7fa54403a7f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 --2- 172.21.15.39:0/2703010865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa558106f00 0x7fa55819a110 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 >> 172.21.15.39:0/2703010865 conn(0x7fa5581008d0 msgr2=0x7fa558101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 shutdown_connections 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:28.025+0000 7fa55f78f700 1 -- 172.21.15.39:0/2703010865 wait complete. 2024-02-19T18:55:28.647 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying grafana service with default placement... 2024-02-19T18:55:29.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:28 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:29.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:28 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:29.936 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-19T18:55:29.936 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfecf6b700 1 Processor -- start 2024-02-19T18:55:29.936 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfecf6b700 1 -- start start 2024-02-19T18:55:29.936 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfecf6b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe8109370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:29.936 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfecf6b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe8079e00 con 0x7fdfe8106f80 2024-02-19T18:55:29.936 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfe659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe8109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfe659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe8109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41928/0 (socket says 172.21.15.39:41928) 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfe659c700 1 -- 172.21.15.39:0/4180585775 learned_addr learned my addr 172.21.15.39:0/4180585775 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.059+0000 7fdfe659c700 1 -- 172.21.15.39:0/4180585775 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe8079f40 con 0x7fdfe8106f80 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfe659c700 1 --2- 172.21.15.39:0/4180585775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe8109370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80098a0 tx=0x7fdfd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffe1eb0e92b2d49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfe559a700 1 -- 172.21.15.39:0/4180585775 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd8004180 con 0x7fdfe8106f80 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfe559a700 1 -- 172.21.15.39:0/4180585775 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdfd80042e0 con 0x7fdfe8106f80 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfe559a700 1 -- 172.21.15.39:0/4180585775 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd8011af0 con 0x7fdfe8106f80 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfecf6b700 1 -- 172.21.15.39:0/4180585775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 msgr2=0x7fdfe8109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfecf6b700 1 --2- 172.21.15.39:0/4180585775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe8109370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80098a0 tx=0x7fdfd8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfecf6b700 1 -- 172.21.15.39:0/4180585775 shutdown_connections 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfecf6b700 1 --2- 172.21.15.39:0/4180585775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe8109370 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.060+0000 7fdfecf6b700 1 -- 172.21.15.39:0/4180585775 >> 172.21.15.39:0/4180585775 conn(0x7fdfe81008d0 msgr2=0x7fdfe8102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.061+0000 7fdfecf6b700 1 -- 172.21.15.39:0/4180585775 shutdown_connections 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.061+0000 7fdfecf6b700 1 -- 172.21.15.39:0/4180585775 wait complete. 2024-02-19T18:55:29.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.061+0000 7fdfecf6b700 1 Processor -- start 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.061+0000 7fdfecf6b700 1 -- start start 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.061+0000 7fdfecf6b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.061+0000 7fdfecf6b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe819a650 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfe659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfe659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41940/0 (socket says 172.21.15.39:41940) 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfe659c700 1 -- 172.21.15.39:0/892459469 learned_addr learned my addr 172.21.15.39:0/892459469 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfe659c700 1 -- 172.21.15.39:0/892459469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd8009550 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfe659c700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe819a110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8011670 tx=0x7fdfd8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd8003a40 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdfd8003ba0 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe819a850 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd8024790 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.062+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe819acf0 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.063+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fdfd802d3f0 con 0x7fdfe8106f80 2024-02-19T18:55:29.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.063+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe810a5d0 con 0x7fdfe8106f80 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.064+0000 7fdfd77fe700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdfd0038370 0x7fdfd003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.064+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdfd804c400 con 0x7fdfe8106f80 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.064+0000 7fdfe5d9b700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdfd0038370 0x7fdfd003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.064+0000 7fdfe5d9b700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdfd0038370 0x7fdfd003a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc006fd0 tx=0x7fdfdc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.069+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdfd8010bd0 con 0x7fdfe8106f80 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.283+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe819db30 con 0x7fdfd0038370 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.287+0000 7fdfd77fe700 1 -- 172.21.15.39:0/892459469 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fdfe819db30 con 0x7fdfd0038370 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdfd0038370 msgr2=0x7fdfd003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdfd0038370 0x7fdfd003a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc006fd0 tx=0x7fdfdc006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 msgr2=0x7fdfe819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe819a110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8011670 tx=0x7fdfd8011b50 comp rx=0 tx=0).stop 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 shutdown_connections 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdfd0038370 0x7fdfd003a830 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 --2- 172.21.15.39:0/892459469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfe8106f80 0x7fdfe819a110 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:29.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.289+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 >> 172.21.15.39:0/892459469 conn(0x7fdfe81008d0 msgr2=0x7fdfe81016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:29.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.290+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 shutdown_connections 2024-02-19T18:55:29.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:29.290+0000 7fdfecf6b700 1 -- 172.21.15.39:0/892459469 wait complete. 2024-02-19T18:55:29.940 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying node-exporter service with default placement... 2024-02-19T18:55:30.231 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:29 smithi039 ceph-mon[28988]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:30.231 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:29 smithi039 ceph-mon[28988]: Saving service prometheus spec with placement count:1 2024-02-19T18:55:30.231 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:29 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:31.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:30 smithi039 ceph-mon[28988]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:31.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:30 smithi039 ceph-mon[28988]: Saving service grafana spec with placement count:1 2024-02-19T18:55:31.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:30 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.316+0000 7f0534035700 1 Processor -- start 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0534035700 1 -- start start 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0534035700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0534035700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c106e10 con 0x7f052c106420 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0531dd1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0531dd1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41954/0 (socket says 172.21.15.39:41954) 2024-02-19T18:55:31.623 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0531dd1700 1 -- 172.21.15.39:0/1061901667 learned_addr learned my addr 172.21.15.39:0/1061901667 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0531dd1700 1 -- 172.21.15.39:0/1061901667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c107620 con 0x7f052c106420 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0531dd1700 1 --2- 172.21.15.39:0/1061901667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c106840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f05200098a0 tx=0x7f0520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e775c32e2b0273d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.317+0000 7f0530dcf700 1 -- 172.21.15.39:0/1061901667 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0520004180 con 0x7f052c106420 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0530dcf700 1 -- 172.21.15.39:0/1061901667 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f05200042e0 con 0x7f052c106420 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0530dcf700 1 -- 172.21.15.39:0/1061901667 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0520011af0 con 0x7f052c106420 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 -- 172.21.15.39:0/1061901667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 msgr2=0x7f052c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 --2- 172.21.15.39:0/1061901667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c106840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f05200098a0 tx=0x7f0520009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 -- 172.21.15.39:0/1061901667 shutdown_connections 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 --2- 172.21.15.39:0/1061901667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c106840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 -- 172.21.15.39:0/1061901667 >> 172.21.15.39:0/1061901667 conn(0x7f052c101a00 msgr2=0x7f052c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 -- 172.21.15.39:0/1061901667 shutdown_connections 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.318+0000 7f0534035700 1 -- 172.21.15.39:0/1061901667 wait complete. 2024-02-19T18:55:31.624 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0534035700 1 Processor -- start 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0534035700 1 -- start start 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0534035700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c19a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0534035700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c19a680 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0531dd1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c19a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0531dd1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c19a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41956/0 (socket says 172.21.15.39:41956) 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0531dd1700 1 -- 172.21.15.39:0/3697466992 learned_addr learned my addr 172.21.15.39:0/3697466992 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.319+0000 7f0531dd1700 1 -- 172.21.15.39:0/3697466992 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0520009550 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.320+0000 7f0531dd1700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c19a140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0520000c00 tx=0x7f0520011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.320+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0520003a70 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.320+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c19a880 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.320+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0520003bd0 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.320+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05200238c0 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.320+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c19ad20 con 0x7f052c106420 2024-02-19T18:55:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.321+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0520023b80 con 0x7f052c106420 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.321+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c0623c0 con 0x7f052c106420 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.321+0000 7f051effd700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0518038320 0x7f051803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.321+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f052004c2c0 con 0x7f052c106420 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.321+0000 7f05315d0700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0518038320 0x7f051803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.321+0000 7f05315d0700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0518038320 0x7f051803a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0528006fd0 tx=0x7f0528006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.326+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0520010970 con 0x7f052c106420 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.537+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f052c19db10 con 0x7f0518038320 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.541+0000 7f051effd700 1 -- 172.21.15.39:0/3697466992 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f052c19db10 con 0x7f0518038320 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0518038320 msgr2=0x7f051803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0518038320 0x7f051803a7e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0528006fd0 tx=0x7f0528006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 msgr2=0x7f052c19a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c19a140 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0520000c00 tx=0x7f0520011b50 comp rx=0 tx=0).stop 2024-02-19T18:55:31.626 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 shutdown_connections 2024-02-19T18:55:31.627 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f0518038320 0x7f051803a7e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.627 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 --2- 172.21.15.39:0/3697466992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f052c106420 0x7f052c19a140 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.627 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 >> 172.21.15.39:0/3697466992 conn(0x7f052c101a00 msgr2=0x7f052c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:31.627 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 shutdown_connections 2024-02-19T18:55:31.627 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:30.543+0000 7f0534035700 1 -- 172.21.15.39:0/3697466992 wait complete. 2024-02-19T18:55:31.627 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying alertmanager service with default placement... 2024-02-19T18:55:32.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:31 smithi039 ceph-mon[28988]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:32.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:31 smithi039 ceph-mon[28988]: Saving service node-exporter spec with placement * 2024-02-19T18:55:32.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:31 smithi039 ceph-mon[28988]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffacd73a700 1 Processor -- start 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffacd73a700 1 -- start start 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffacd73a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffacd73a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8079e60 con 0x7ffac8106f00 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffac6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffac6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41962/0 (socket says 172.21.15.39:41962) 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.045+0000 7ffac6ffd700 1 -- 172.21.15.39:0/2209949619 learned_addr learned my addr 172.21.15.39:0/2209949619 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffac6ffd700 1 -- 172.21.15.39:0/2209949619 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8109830 con 0x7ffac8106f00 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffac6ffd700 1 --2- 172.21.15.39:0/2209949619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac81092f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffab80098a0 tx=0x7ffab8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e8aacb24a900de server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffac5ffb700 1 -- 172.21.15.39:0/2209949619 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8004180 con 0x7ffac8106f00 2024-02-19T18:55:32.837 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffac5ffb700 1 -- 172.21.15.39:0/2209949619 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffab80042e0 con 0x7ffac8106f00 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffac5ffb700 1 -- 172.21.15.39:0/2209949619 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8011af0 con 0x7ffac8106f00 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffacd73a700 1 -- 172.21.15.39:0/2209949619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 msgr2=0x7ffac81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffacd73a700 1 --2- 172.21.15.39:0/2209949619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac81092f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffab80098a0 tx=0x7ffab8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffacd73a700 1 -- 172.21.15.39:0/2209949619 shutdown_connections 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.046+0000 7ffacd73a700 1 --2- 172.21.15.39:0/2209949619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac81092f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.047+0000 7ffacd73a700 1 -- 172.21.15.39:0/2209949619 >> 172.21.15.39:0/2209949619 conn(0x7ffac81008b0 msgr2=0x7ffac8102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.047+0000 7ffacd73a700 1 -- 172.21.15.39:0/2209949619 shutdown_connections 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.047+0000 7ffacd73a700 1 -- 172.21.15.39:0/2209949619 wait complete. 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.047+0000 7ffacd73a700 1 Processor -- start 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.047+0000 7ffacd73a700 1 -- start start 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffacd73a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac8106860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffacd73a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8106da0 con 0x7ffac8106f00 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffac6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac8106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:32.838 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffac6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac8106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41974/0 (socket says 172.21.15.39:41974) 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffac6ffd700 1 -- 172.21.15.39:0/2932719620 learned_addr learned my addr 172.21.15.39:0/2932719620 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffac6ffd700 1 -- 172.21.15.39:0/2932719620 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8009550 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffac6ffd700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac8106860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffab8011670 tx=0x7ffab8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8003a40 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffab8003ba0 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac8104eb0 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab8024820 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.048+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac8105350 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.049+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ffab8024ae0 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.049+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac810a5c0 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.049+0000 7ffaaffff700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ffab00382e0 0x7ffab003a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.050+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffab804c490 con 0x7ffac8106f00 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.050+0000 7ffac67fc700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ffab00382e0 0x7ffab003a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:32.839 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.050+0000 7ffac67fc700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ffab00382e0 0x7ffab003a7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006fd0 tx=0x7ffabc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.054+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffab8024d90 con 0x7ffac8106f00 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.263+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7ffac8101dc0 con 0x7ffab00382e0 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.267+0000 7ffaaffff700 1 -- 172.21.15.39:0/2932719620 <== mgr.14120 v2:172.21.15.39:6800/2039816448 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7ffac8101dc0 con 0x7ffab00382e0 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ffab00382e0 msgr2=0x7ffab003a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ffab00382e0 0x7ffab003a7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006fd0 tx=0x7ffabc006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 msgr2=0x7ffac8106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac8106860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffab8011670 tx=0x7ffab8011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 shutdown_connections 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7ffab00382e0 0x7ffab003a7a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 --2- 172.21.15.39:0/2932719620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffac8106f00 0x7ffac8106860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 >> 172.21.15.39:0/2932719620 conn(0x7ffac81008b0 msgr2=0x7ffac81016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 shutdown_connections 2024-02-19T18:55:32.840 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:32.269+0000 7ffacd73a700 1 -- 172.21.15.39:0/2932719620 wait complete. 2024-02-19T18:55:33.435 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:33 smithi039 ceph-mon[28988]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:33.435 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:33 smithi039 ceph-mon[28988]: Saving service alertmanager spec with placement count:1 2024-02-19T18:55:33.436 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:33 smithi039 ceph-mon[28988]: from='mgr.14120 172.21.15.39:0/3576023007' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:33.436 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:33 smithi039 ceph-mon[28988]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:34.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.210+0000 7f832c94c700 1 Processor -- start 2024-02-19T18:55:34.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832c94c700 1 -- start start 2024-02-19T18:55:34.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832c94c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f8328106d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:34.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832c94c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83281072e0 con 0x7f83281068f0 2024-02-19T18:55:34.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f8328106d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:34.236 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f8328106d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41984/0 (socket says 172.21.15.39:41984) 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832659c700 1 -- 172.21.15.39:0/2201549518 learned_addr learned my addr 172.21.15.39:0/2201549518 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832659c700 1 -- 172.21.15.39:0/2201549518 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328105550 con 0x7f83281068f0 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.211+0000 7f832659c700 1 --2- 172.21.15.39:0/2201549518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f8328106d10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f83180098a0 tx=0x7f8318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ad331e313a6cb0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832559a700 1 -- 172.21.15.39:0/2201549518 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8318004180 con 0x7f83281068f0 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832559a700 1 -- 172.21.15.39:0/2201549518 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f83180042e0 con 0x7f83281068f0 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832559a700 1 -- 172.21.15.39:0/2201549518 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8318011af0 con 0x7f83281068f0 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 -- 172.21.15.39:0/2201549518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 msgr2=0x7f8328106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 --2- 172.21.15.39:0/2201549518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f8328106d10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f83180098a0 tx=0x7f8318009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 -- 172.21.15.39:0/2201549518 shutdown_connections 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 --2- 172.21.15.39:0/2201549518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f8328106d10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 -- 172.21.15.39:0/2201549518 >> 172.21.15.39:0/2201549518 conn(0x7f8328100040 msgr2=0x7f83281024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 -- 172.21.15.39:0/2201549518 shutdown_connections 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.212+0000 7f832c94c700 1 -- 172.21.15.39:0/2201549518 wait complete. 2024-02-19T18:55:34.237 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832c94c700 1 Processor -- start 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832c94c700 1 -- start start 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832c94c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f832819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832c94c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832819a670 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f832819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f832819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41998/0 (socket says 172.21.15.39:41998) 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.213+0000 7f832659c700 1 -- 172.21.15.39:0/1309737690 learned_addr learned my addr 172.21.15.39:0/1309737690 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f832659c700 1 -- 172.21.15.39:0/1309737690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318009550 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f832659c700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f832819a130 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8318011670 tx=0x7f8318011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8318003a40 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8318003ba0 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832819a870 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8318024790 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.214+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832819ac50 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.215+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f831802d3f0 con 0x7f83281068f0 2024-02-19T18:55:34.238 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.215+0000 7f831f7fe700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8314038320 0x7f831403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.215+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f831804c6b0 con 0x7f83281068f0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.215+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83280623c0 con 0x7f83281068f0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.215+0000 7f8325d9b700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8314038320 0x7f831403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.215+0000 7f8325d9b700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8314038320 0x7f831403a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8310006fd0 tx=0x7f8310006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.220+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8318019460 con 0x7f83281068f0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.418+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f832818efb0 con 0x7f83281068f0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.422+0000 7f831f7fe700 1 -- 172.21.15.39:0/1309737690 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f831800b3e0 con 0x7f83281068f0 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8314038320 msgr2=0x7f831403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8314038320 0x7f831403a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8310006fd0 tx=0x7f8310006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 msgr2=0x7f832819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f832819a130 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8318011670 tx=0x7f8318011b50 comp rx=0 tx=0).stop 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 shutdown_connections 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8314038320 0x7f831403a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:34.239 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 --2- 172.21.15.39:0/1309737690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281068f0 0x7f832819a130 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:34.240 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 >> 172.21.15.39:0/1309737690 conn(0x7f8328100040 msgr2=0x7f8328104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:34.240 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 shutdown_connections 2024-02-19T18:55:34.240 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:33.424+0000 7f832c94c700 1 -- 172.21.15.39:0/1309737690 wait complete. 2024-02-19T18:55:34.493 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:34 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1309737690' entity='client.admin' 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f7320bcb700 1 Processor -- start 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f7320bcb700 1 -- start start 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f7320bcb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f7320bcb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c106e10 con 0x7f731c106420 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f731a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f731a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42000/0 (socket says 172.21.15.39:42000) 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f731a59c700 1 -- 172.21.15.39:0/126219316 learned_addr learned my addr 172.21.15.39:0/126219316 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.633+0000 7f731a59c700 1 -- 172.21.15.39:0/126219316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c107620 con 0x7f731c106420 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.634+0000 7f731a59c700 1 --2- 172.21.15.39:0/126219316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c106840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f730c0098a0 tx=0x7f730c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b369f123134572 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.634+0000 7f731959a700 1 -- 172.21.15.39:0/126219316 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730c004180 con 0x7f731c106420 2024-02-19T18:55:35.368 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.634+0000 7f731959a700 1 -- 172.21.15.39:0/126219316 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f730c0042e0 con 0x7f731c106420 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.634+0000 7f731959a700 1 -- 172.21.15.39:0/126219316 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730c011af0 con 0x7f731c106420 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.634+0000 7f7320bcb700 1 -- 172.21.15.39:0/126219316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 msgr2=0x7f731c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.634+0000 7f7320bcb700 1 --2- 172.21.15.39:0/126219316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c106840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f730c0098a0 tx=0x7f730c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 -- 172.21.15.39:0/126219316 shutdown_connections 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 --2- 172.21.15.39:0/126219316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c106840 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 -- 172.21.15.39:0/126219316 >> 172.21.15.39:0/126219316 conn(0x7f731c101a00 msgr2=0x7f731c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 -- 172.21.15.39:0/126219316 shutdown_connections 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 -- 172.21.15.39:0/126219316 wait complete. 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 Processor -- start 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.635+0000 7f7320bcb700 1 -- start start 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f7320bcb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f7320bcb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c19a620 con 0x7f731c106420 2024-02-19T18:55:35.369 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f731a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f731a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42012/0 (socket says 172.21.15.39:42012) 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f731a59c700 1 -- 172.21.15.39:0/3006014934 learned_addr learned my addr 172.21.15.39:0/3006014934 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f731a59c700 1 -- 172.21.15.39:0/3006014934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c009550 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f731a59c700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c19a0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f730c006ba0 tx=0x7f730c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730c003b00 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c19a820 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f730c01a8e0 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730c019990 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.636+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c19acc0 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.637+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f730c019c00 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.637+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c0623c0 con 0x7f731c106420 2024-02-19T18:55:35.370 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.637+0000 7f730b7fe700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f7304038320 0x7f730403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.637+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f730c04c580 con 0x7f731c106420 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.638+0000 7f7319d9b700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f7304038320 0x7f730403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.638+0000 7f7319d9b700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f7304038320 0x7f730403a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7310006fd0 tx=0x7f7310006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.642+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f730c02c3a0 con 0x7f731c106420 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.839+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f731c10a820 con 0x7f731c106420 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.842+0000 7f730b7fe700 1 -- 172.21.15.39:0/3006014934 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f730c0172b0 con 0x7f731c106420 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f7304038320 msgr2=0x7f730403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f7304038320 0x7f730403a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7310006fd0 tx=0x7f7310006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 msgr2=0x7f731c19a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c19a0e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f730c006ba0 tx=0x7f730c011b50 comp rx=0 tx=0).stop 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 shutdown_connections 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f7304038320 0x7f730403a7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 --2- 172.21.15.39:0/3006014934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f731c106420 0x7f731c19a0e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:35.371 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 >> 172.21.15.39:0/3006014934 conn(0x7f731c101a00 msgr2=0x7f731c1027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:35.372 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 shutdown_connections 2024-02-19T18:55:35.372 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:34.845+0000 7f7320bcb700 1 -- 172.21.15.39:0/3006014934 wait complete. 2024-02-19T18:55:35.372 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling the dashboard module... 2024-02-19T18:55:36.035 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:35 smithi039 ceph-mon[28988]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:36.128 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:35 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3006014934' entity='client.admin' 2024-02-19T18:55:37.135 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:36 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/4077508689' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce3e17700 1 Processor -- start 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce3e17700 1 -- start start 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce3e17700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc109330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce3e17700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc079e40 con 0x7fdcdc106f40 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce1bb3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce1bb3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42026/0 (socket says 172.21.15.39:42026) 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.750+0000 7fdce1bb3700 1 -- 172.21.15.39:0/3288475987 learned_addr learned my addr 172.21.15.39:0/3288475987 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.751+0000 7fdce1bb3700 1 -- 172.21.15.39:0/3288475987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc109870 con 0x7fdcdc106f40 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.751+0000 7fdce1bb3700 1 --2- 172.21.15.39:0/3288475987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc109330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098a0 tx=0x7fdccc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8229a0b8ed7860 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.751+0000 7fdce0bb1700 1 -- 172.21.15.39:0/3288475987 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdccc004180 con 0x7fdcdc106f40 2024-02-19T18:55:37.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.751+0000 7fdce0bb1700 1 -- 172.21.15.39:0/3288475987 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdccc0042e0 con 0x7fdcdc106f40 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.751+0000 7fdce0bb1700 1 -- 172.21.15.39:0/3288475987 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdccc011af0 con 0x7fdcdc106f40 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 -- 172.21.15.39:0/3288475987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 msgr2=0x7fdcdc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 --2- 172.21.15.39:0/3288475987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc109330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098a0 tx=0x7fdccc009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 -- 172.21.15.39:0/3288475987 shutdown_connections 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 --2- 172.21.15.39:0/3288475987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc109330 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 -- 172.21.15.39:0/3288475987 >> 172.21.15.39:0/3288475987 conn(0x7fdcdc1008d0 msgr2=0x7fdcdc102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 -- 172.21.15.39:0/3288475987 shutdown_connections 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 -- 172.21.15.39:0/3288475987 wait complete. 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 Processor -- start 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.752+0000 7fdce3e17700 1 -- start start 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce3e17700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce3e17700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcdc19a670 con 0x7fdcdc106f40 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce1bb3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:37.460 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce1bb3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42040/0 (socket says 172.21.15.39:42040) 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce1bb3700 1 -- 172.21.15.39:0/4077508689 learned_addr learned my addr 172.21.15.39:0/4077508689 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce1bb3700 1 -- 172.21.15.39:0/4077508689 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc009550 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce1bb3700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc19a130 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdccc011670 tx=0x7fdccc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdccc003a40 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdccc003ba0 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc19a870 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdccc024820 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.753+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc19ad10 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.754+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fdccc01ac70 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.754+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc10a5b0 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.755+0000 7fdcd2ffd700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdcc8038330 0x7fdcc803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.755+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdccc04c2c0 con 0x7fdcdc106f40 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.755+0000 7fdce13b2700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdcc8038330 0x7fdcc803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:37.461 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.755+0000 7fdce13b2700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdcc8038330 0x7fdcc803a7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8006fd0 tx=0x7fdcd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:35.759+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdccc018760 con 0x7fdcdc106f40 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.018+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fdcdc10a7d0 con 0x7fdcdc106f40 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.849+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fdccc024980 con 0x7fdcdc106f40 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.850+0000 7fdcd2ffd700 1 -- 172.21.15.39:0/4077508689 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fdccc01a950 con 0x7fdcdc106f40 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.852+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdcc8038330 msgr2=0x7fdcc803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.852+0000 7fdce3e17700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdcc8038330 0x7fdcc803a7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8006fd0 tx=0x7fdcd8006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 msgr2=0x7fdcdc19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc19a130 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdccc011670 tx=0x7fdccc011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 shutdown_connections 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fdcc8038330 0x7fdcc803a7f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 --2- 172.21.15.39:0/4077508689 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcdc106f40 0x7fdcdc19a130 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 >> 172.21.15.39:0/4077508689 conn(0x7fdcdc1008d0 msgr2=0x7fdcdc1016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 shutdown_connections 2024-02-19T18:55:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:36.853+0000 7fdce3e17700 1 -- 172.21.15.39:0/4077508689 wait complete. 2024-02-19T18:55:38.052 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:37 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/4077508689' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T18:55:38.052 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:37 smithi039 ceph-mon[28988]: mgrmap e9: smithi039.ydmpuh(active, since 26s) 2024-02-19T18:55:38.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:55:38.733 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "smithi039.ydmpuh", 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876fb87700 1 Processor -- start 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876fb87700 1 -- start start 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876fb87700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f8768104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876fb87700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8768105120 con 0x7f8768104730 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876d923700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f8768104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876d923700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f8768104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48456/0 (socket says 172.21.15.39:48456) 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.830+0000 7f876d923700 1 -- 172.21.15.39:0/1123373485 learned_addr learned my addr 172.21.15.39:0/1123373485 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.831+0000 7f876d923700 1 -- 172.21.15.39:0/1123373485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87681076c0 con 0x7f8768104730 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.831+0000 7f876d923700 1 --2- 172.21.15.39:0/1123373485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f8768104b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8758009e80 tx=0x7f875800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45b6773a85f6c33 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:38.734 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.831+0000 7f876c921700 1 -- 172.21.15.39:0/1123373485 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f875800e070 con 0x7f8768104730 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.831+0000 7f876c921700 1 -- 172.21.15.39:0/1123373485 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f875800b8c0 con 0x7f8768104730 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.831+0000 7f876c921700 1 -- 172.21.15.39:0/1123373485 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87580044d0 con 0x7f8768104730 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 -- 172.21.15.39:0/1123373485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 msgr2=0x7f8768104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 --2- 172.21.15.39:0/1123373485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f8768104b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8758009e80 tx=0x7f875800b330 comp rx=0 tx=0).stop 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 -- 172.21.15.39:0/1123373485 shutdown_connections 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 --2- 172.21.15.39:0/1123373485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f8768104b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 -- 172.21.15.39:0/1123373485 >> 172.21.15.39:0/1123373485 conn(0x7f8768100040 msgr2=0x7f87681024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 -- 172.21.15.39:0/1123373485 shutdown_connections 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 -- 172.21.15.39:0/1123373485 wait complete. 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 Processor -- start 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.832+0000 7f876fb87700 1 -- start start 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876fb87700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f876819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876fb87700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876819a6a0 con 0x7f8768104730 2024-02-19T18:55:38.735 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876d923700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f876819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876d923700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f876819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48472/0 (socket says 172.21.15.39:48472) 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876d923700 1 -- 172.21.15.39:0/1122552217 learned_addr learned my addr 172.21.15.39:0/1122552217 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876d923700 1 -- 172.21.15.39:0/1122552217 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758009b30 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876d923700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f876819a160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8758009e50 tx=0x7f87580049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f875800e040 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f875800bcc0 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8758012650 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876819a8a0 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.833+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876819ada0 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.834+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f8758019040 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.835+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680623c0 con 0x7f8768104730 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.835+0000 7f8766ffd700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 0x7f875003a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:38.736 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.835+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f875804be50 con 0x7f8768104730 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.835+0000 7f876d122700 1 -- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 msgr2=0x7f875003a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.835+0000 7f876d122700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 0x7f875003a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:37.840+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8758011750 con 0x7f8768104730 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.036+0000 7f876d122700 1 -- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 msgr2=0x7f875003a840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.036+0000 7f876d122700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 0x7f875003a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.100+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f876810a8b0 con 0x7f8768104730 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.100+0000 7f8766ffd700 1 -- 172.21.15.39:0/1122552217 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f875804d080 con 0x7f8768104730 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.102+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 msgr2=0x7f875003a840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.102+0000 7f876fb87700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 0x7f875003a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.102+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 msgr2=0x7f876819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.102+0000 7f876fb87700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f876819a160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8758009e50 tx=0x7f87580049f0 comp rx=0 tx=0).stop 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.103+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 shutdown_connections 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.103+0000 7f876fb87700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7f8750038380 0x7f875003a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:38.737 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.103+0000 7f876fb87700 1 --2- 172.21.15.39:0/1122552217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8768104730 0x7f876819a160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:38.738 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.103+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 >> 172.21.15.39:0/1122552217 conn(0x7f8768100040 msgr2=0x7f8768101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:38.738 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.103+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 shutdown_connections 2024-02-19T18:55:38.738 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:38.103+0000 7f876fb87700 1 -- 172.21.15.39:0/1122552217 wait complete. 2024-02-19T18:55:38.738 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2024-02-19T18:55:38.738 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 9... 2024-02-19T18:55:38.975 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:38 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1122552217' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: Active manager daemon smithi039.ydmpuh restarted 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: Activating manager daemon smithi039.ydmpuh 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: osdmap e3: 0 total, 0 up, 0 in 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: mgrmap e10: smithi039.ydmpuh(active, starting, since 0.00354425s) 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:43.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:43 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:44.922 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:44 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:55:44.922 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:44 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T18:55:44.922 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:44 smithi039 ceph-mon[28988]: mgrmap e11: smithi039.ydmpuh(active, since 1.00556s) 2024-02-19T18:55:45.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.179+0000 7fac09494700 1 Processor -- start 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac09494700 1 -- start start 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac09494700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac04106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac09494700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac04106e10 con 0x7fac04106420 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac02ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac04106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac02ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac04106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48480/0 (socket says 172.21.15.39:48480) 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac02ffd700 1 -- 172.21.15.39:0/2636530264 learned_addr learned my addr 172.21.15.39:0/2636530264 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac02ffd700 1 -- 172.21.15.39:0/2636530264 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04107620 con 0x7fac04106420 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.180+0000 7fac02ffd700 1 --2- 172.21.15.39:0/2636530264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac04106840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098a0 tx=0x7fabf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02442b558a19f5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac01ffb700 1 -- 172.21.15.39:0/2636530264 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabf8004180 con 0x7fac04106420 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac01ffb700 1 -- 172.21.15.39:0/2636530264 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fabf80042e0 con 0x7fac04106420 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac01ffb700 1 -- 172.21.15.39:0/2636530264 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabf8011af0 con 0x7fac04106420 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 -- 172.21.15.39:0/2636530264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 msgr2=0x7fac04106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 --2- 172.21.15.39:0/2636530264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac04106840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098a0 tx=0x7fabf8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 -- 172.21.15.39:0/2636530264 shutdown_connections 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 --2- 172.21.15.39:0/2636530264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac04106840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 -- 172.21.15.39:0/2636530264 >> 172.21.15.39:0/2636530264 conn(0x7fac04101a00 msgr2=0x7fac04103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 -- 172.21.15.39:0/2636530264 shutdown_connections 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.181+0000 7fac09494700 1 -- 172.21.15.39:0/2636530264 wait complete. 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac09494700 1 Processor -- start 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac09494700 1 -- start start 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac09494700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac0419c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac09494700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0419c8d0 con 0x7fac04106420 2024-02-19T18:55:45.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac02ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac0419c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac02ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac0419c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48490/0 (socket says 172.21.15.39:48490) 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.182+0000 7fac02ffd700 1 -- 172.21.15.39:0/2732879503 learned_addr learned my addr 172.21.15.39:0/2732879503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fac02ffd700 1 -- 172.21.15.39:0/2732879503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8009550 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fac02ffd700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac0419c390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabf8011670 tx=0x7fabf8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabf80039c0 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fabf8003b20 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0419cad0 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabf80247f0 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.183+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0419cf70 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.184+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fabf8024ab0 con 0x7fac04106420 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.184+0000 7fabebfff700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.184+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 --> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabec03af00 con 0x7fabec038330 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.184+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fabf804c480 con 0x7fac04106420 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.184+0000 7fac027fc700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 msgr2=0x7fabec03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.184+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.385+0000 7fac027fc700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 msgr2=0x7fabec03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.385+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.786+0000 7fac027fc700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 msgr2=0x7fabec03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:39.786+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:40.587+0000 7fac027fc700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 msgr2=0x7fabec03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:40.587+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:42.188+0000 7fac027fc700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 msgr2=0x7fabec03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2039816448 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:42.188+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:43.706+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7fabf801a8e0 con 0x7fac04106420 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:43.706+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 msgr2=0x7fabec03a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:55:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:43.706+0000 7fabebfff700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.707+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7fabf804dd10 con 0x7fac04106420 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.707+0000 7fabebfff700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fabec03bdf0 0x7fabec03e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.707+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabec03af00 con 0x7fabec03bdf0 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.708+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fabec03bdf0 0x7fabec03e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.708+0000 7fac027fc700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fabec03bdf0 0x7fabec03e1e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fabf4003a10 tx=0x7fabf40092b0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.709+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fabec03af00 con 0x7fabec03bdf0 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.715+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fac0419d6d0 con 0x7fabec03bdf0 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.715+0000 7fabebfff700 1 -- 172.21.15.39:0/2732879503 <== mgr.14164 v2:172.21.15.39:6800/2158365942 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fac0419d6d0 con 0x7fabec03bdf0 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.715+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fabec03bdf0 msgr2=0x7fabec03e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fabec03bdf0 0x7fabec03e1e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fabf4003a10 tx=0x7fabf40092b0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 msgr2=0x7fac0419c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:45.474 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac0419c390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabf8011670 tx=0x7fabf8011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 shutdown_connections 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fabec03bdf0 0x7fabec03e1e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac04106420 0x7fac0419c390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 --2- 172.21.15.39:0/2732879503 >> [v2:172.21.15.39:6800/2039816448,v1:172.21.15.39:6801/2039816448] conn(0x7fabec038330 0x7fabec03a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 >> 172.21.15.39:0/2732879503 conn(0x7fac04101a00 msgr2=0x7fac041030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 shutdown_connections 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:44.716+0000 7fac09494700 1 -- 172.21.15.39:0/2732879503 wait complete. 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 9 is available 2024-02-19T18:55:45.475 INFO:teuthology.orchestra.run.smithi039.stdout:Generating a dashboard self-signed certificate... 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:44] ENGINE Bus STARTING 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:44] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:44] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: [19/Feb/2024:18:55:44] ENGINE Bus STARTED 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T18:55:46.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:45 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.848+0000 7fd64ad58700 1 Processor -- start 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.848+0000 7fd64ad58700 1 -- start start 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.848+0000 7fd64ad58700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd644106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.848+0000 7fd64ad58700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644106e10 con 0x7fd644106420 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd648af4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd644106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd648af4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd644106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48562/0 (socket says 172.21.15.39:48562) 2024-02-19T18:55:46.884 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd648af4700 1 -- 172.21.15.39:0/3901086252 learned_addr learned my addr 172.21.15.39:0/3901086252 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd648af4700 1 -- 172.21.15.39:0/3901086252 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd644107620 con 0x7fd644106420 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd648af4700 1 --2- 172.21.15.39:0/3901086252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd644106840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6380098a0 tx=0x7fd638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7686a21fa7e2ff13 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd6437fe700 1 -- 172.21.15.39:0/3901086252 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd638004180 con 0x7fd644106420 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd6437fe700 1 -- 172.21.15.39:0/3901086252 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd6380042e0 con 0x7fd644106420 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.849+0000 7fd6437fe700 1 -- 172.21.15.39:0/3901086252 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd638011af0 con 0x7fd644106420 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 -- 172.21.15.39:0/3901086252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 msgr2=0x7fd644106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 --2- 172.21.15.39:0/3901086252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd644106840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd6380098a0 tx=0x7fd638009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 -- 172.21.15.39:0/3901086252 shutdown_connections 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 --2- 172.21.15.39:0/3901086252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd644106840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 -- 172.21.15.39:0/3901086252 >> 172.21.15.39:0/3901086252 conn(0x7fd644101a00 msgr2=0x7fd644103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 -- 172.21.15.39:0/3901086252 shutdown_connections 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.850+0000 7fd64ad58700 1 -- 172.21.15.39:0/3901086252 wait complete. 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd64ad58700 1 Processor -- start 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd64ad58700 1 -- start start 2024-02-19T18:55:46.885 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd64ad58700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd64419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd64ad58700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64419a6c0 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd648af4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd64419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd648af4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd64419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48568/0 (socket says 172.21.15.39:48568) 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd648af4700 1 -- 172.21.15.39:0/2991712825 learned_addr learned my addr 172.21.15.39:0/2991712825 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd648af4700 1 -- 172.21.15.39:0/2991712825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638009550 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd648af4700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd64419a180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd638011670 tx=0x7fd638011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.851+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6380039c0 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.852+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd638003b20 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.852+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64419a8c0 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.852+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6380247f0 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.852+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64419ad60 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.853+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7fd638024ab0 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.853+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6440623c0 con 0x7fd644106420 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.853+0000 7fd641ffb700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd634038210 0x7fd63403a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:46.886 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.853+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd63804c260 con 0x7fd644106420 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.853+0000 7fd643fff700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd634038210 0x7fd63403a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.853+0000 7fd643fff700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd634038210 0x7fd63403a6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd630006fd0 tx=0x7fd630006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.858+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd638022310 con 0x7fd644106420 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:45.976+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fd63801ac70 con 0x7fd644106420 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.061+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd644103d10 con 0x7fd634038210 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.320+0000 7fd641ffb700 1 -- 172.21.15.39:0/2991712825 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd644103d10 con 0x7fd634038210 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd634038210 msgr2=0x7fd63403a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd634038210 0x7fd63403a6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd630006fd0 tx=0x7fd630006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 msgr2=0x7fd64419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd64419a180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd638011670 tx=0x7fd638011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 shutdown_connections 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd634038210 0x7fd63403a6d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 --2- 172.21.15.39:0/2991712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd644106420 0x7fd64419a180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.887 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 >> 172.21.15.39:0/2991712825 conn(0x7fd644101a00 msgr2=0x7fd6441030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:46.888 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 shutdown_connections 2024-02-19T18:55:46.888 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:46.323+0000 7fd64ad58700 1 -- 172.21.15.39:0/2991712825 wait complete. 2024-02-19T18:55:46.888 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial admin user... 2024-02-19T18:55:47.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:46 smithi039 ceph-mon[28988]: mgrmap e12: smithi039.ydmpuh(active, since 2s) 2024-02-19T18:55:47.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:46 smithi039 ceph-mon[28988]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:47.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:46 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:47.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:46 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:47.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:46 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:48.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:48 smithi039 ceph-mon[28988]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:48.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:48 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zzBHmn14pqIgKBm1f8Au/e6MdVcIyl8X.US93hkm.jF8wn.WdCuvm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708368947, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb045047700 1 Processor -- start 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb045047700 1 -- start start 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb045047700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb040106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb045047700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040106e10 con 0x7fb040106420 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb03ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb040106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb03ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb040106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53134/0 (socket says 172.21.15.39:53134) 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb03ed9d700 1 -- 172.21.15.39:0/1283943007 learned_addr learned my addr 172.21.15.39:0/1283943007 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.266+0000 7fb03ed9d700 1 -- 172.21.15.39:0/1283943007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb040107620 con 0x7fb040106420 2024-02-19T18:55:49.065 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb03ed9d700 1 --2- 172.21.15.39:0/1283943007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb040106840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098a0 tx=0x7fb030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73fc397b9886d29 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb03dd9b700 1 -- 172.21.15.39:0/1283943007 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030004180 con 0x7fb040106420 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb03dd9b700 1 -- 172.21.15.39:0/1283943007 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb0300042e0 con 0x7fb040106420 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb03dd9b700 1 -- 172.21.15.39:0/1283943007 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030011af0 con 0x7fb040106420 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 -- 172.21.15.39:0/1283943007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 msgr2=0x7fb040106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 --2- 172.21.15.39:0/1283943007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb040106840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098a0 tx=0x7fb030009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 -- 172.21.15.39:0/1283943007 shutdown_connections 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 --2- 172.21.15.39:0/1283943007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb040106840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 -- 172.21.15.39:0/1283943007 >> 172.21.15.39:0/1283943007 conn(0x7fb040101a00 msgr2=0x7fb040103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 -- 172.21.15.39:0/1283943007 shutdown_connections 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.267+0000 7fb045047700 1 -- 172.21.15.39:0/1283943007 wait complete. 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.268+0000 7fb045047700 1 Processor -- start 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.268+0000 7fb045047700 1 -- start start 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.268+0000 7fb045047700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb04019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.268+0000 7fb045047700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04019a670 con 0x7fb040106420 2024-02-19T18:55:49.066 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb03ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb04019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb03ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb04019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53142/0 (socket says 172.21.15.39:53142) 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb03ed9d700 1 -- 172.21.15.39:0/506294480 learned_addr learned my addr 172.21.15.39:0/506294480 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb03ed9d700 1 -- 172.21.15.39:0/506294480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030009550 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb03ed9d700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb04019a130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb030006ba0 tx=0x7fb030011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030003b00 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04019a870 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb03001a8e0 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030019990 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.269+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04019ad10 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.270+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fb030019c50 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.270+0000 7fb027fff700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fb028038320 0x7fb02803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.270+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb03004c760 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.270+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0400623c0 con 0x7fb040106420 2024-02-19T18:55:49.067 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.270+0000 7fb03e59c700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fb028038320 0x7fb02803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.271+0000 7fb03e59c700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fb028038320 0x7fb02803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb034006fd0 tx=0x7fb034006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.275+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb030048080 con 0x7fb040106420 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.493+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb04019db20 con 0x7fb028038320 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.739+0000 7fb027fff700 1 -- 172.21.15.39:0/506294480 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb04019db20 con 0x7fb028038320 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fb028038320 msgr2=0x7fb02803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fb028038320 0x7fb02803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb034006fd0 tx=0x7fb034006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 msgr2=0x7fb04019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb04019a130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb030006ba0 tx=0x7fb030011b50 comp rx=0 tx=0).stop 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 shutdown_connections 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fb028038320 0x7fb02803a7e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 --2- 172.21.15.39:0/506294480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040106420 0x7fb04019a130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 >> 172.21.15.39:0/506294480 conn(0x7fb040101a00 msgr2=0x7fb0401030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:49.068 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 shutdown_connections 2024-02-19T18:55:49.069 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:47.741+0000 7fb045047700 1 -- 172.21.15.39:0/506294480 wait complete. 2024-02-19T18:55:49.069 INFO:teuthology.orchestra.run.smithi039.stdout:Fetching dashboard port number... 2024-02-19T18:55:50.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:49 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/4011431080' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 8443 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8e5d91700 1 Processor -- start 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8e5d91700 1 -- start start 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8e5d91700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e0106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8e5d91700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e0106e10 con 0x7ff8e0106420 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53156/0 (socket says 172.21.15.39:53156) 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8df7fe700 1 -- 172.21.15.39:0/866388087 learned_addr learned my addr 172.21.15.39:0/866388087 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.437+0000 7ff8df7fe700 1 -- 172.21.15.39:0/866388087 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e0107620 con 0x7ff8e0106420 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8df7fe700 1 --2- 172.21.15.39:0/866388087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e0106840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009e80 tx=0x7ff8d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f3e0edee9a2e96 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8de7fc700 1 -- 172.21.15.39:0/866388087 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d000e070 con 0x7ff8e0106420 2024-02-19T18:55:50.224 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8de7fc700 1 -- 172.21.15.39:0/866388087 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff8d000b8c0 con 0x7ff8e0106420 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8de7fc700 1 -- 172.21.15.39:0/866388087 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d00044d0 con 0x7ff8e0106420 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 -- 172.21.15.39:0/866388087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 msgr2=0x7ff8e0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 --2- 172.21.15.39:0/866388087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e0106840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009e80 tx=0x7ff8d000b330 comp rx=0 tx=0).stop 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 -- 172.21.15.39:0/866388087 shutdown_connections 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 --2- 172.21.15.39:0/866388087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e0106840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 -- 172.21.15.39:0/866388087 >> 172.21.15.39:0/866388087 conn(0x7ff8e0101a00 msgr2=0x7ff8e0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 -- 172.21.15.39:0/866388087 shutdown_connections 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.438+0000 7ff8e5d91700 1 -- 172.21.15.39:0/866388087 wait complete. 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8e5d91700 1 Processor -- start 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8e5d91700 1 -- start start 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8e5d91700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8e5d91700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e019a5f0 con 0x7ff8e0106420 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53166/0 (socket says 172.21.15.39:53166) 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.439+0000 7ff8df7fe700 1 -- 172.21.15.39:0/4011431080 learned_addr learned my addr 172.21.15.39:0/4011431080 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8df7fe700 1 -- 172.21.15.39:0/4011431080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0009b30 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8df7fe700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e019a0b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00072a0 tx=0x7ff8d00049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d000e040 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff8d000bd80 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e019a7f0 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d00126c0 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.440+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e019acf0 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.441+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7ff8d0019040 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.441+0000 7ff8dcff9700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8cc038330 0x7ff8cc03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.441+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff8d004be80 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.441+0000 7ff8deffd700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8cc038330 0x7ff8cc03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.441+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e00623c0 con 0x7ff8e0106420 2024-02-19T18:55:50.226 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.441+0000 7ff8deffd700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8cc038330 0x7ff8cc03a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8006fd0 tx=0x7ff8c8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.446+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff8d001f080 con 0x7ff8e0106420 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.644+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff8e019dcc0 con 0x7ff8e0106420 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.645+0000 7ff8dcff9700 1 -- 172.21.15.39:0/4011431080 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff8d004a0b0 con 0x7ff8e0106420 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.646+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8cc038330 msgr2=0x7ff8cc03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.646+0000 7ff8e5d91700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8cc038330 0x7ff8cc03a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8006fd0 tx=0x7ff8c8006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.646+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 msgr2=0x7ff8e019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.646+0000 7ff8e5d91700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e019a0b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8d00072a0 tx=0x7ff8d00049f0 comp rx=0 tx=0).stop 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.647+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 shutdown_connections 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.647+0000 7ff8e5d91700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8cc038330 0x7ff8cc03a7f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.647+0000 7ff8e5d91700 1 --2- 172.21.15.39:0/4011431080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8e0106420 0x7ff8e019a0b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.647+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 >> 172.21.15.39:0/4011431080 conn(0x7ff8e0101a00 msgr2=0x7ff8e01027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.647+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 shutdown_connections 2024-02-19T18:55:50.227 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:49.647+0000 7ff8e5d91700 1 -- 172.21.15.39:0/4011431080 wait complete. 2024-02-19T18:55:50.231 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:55:50.231 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-02-19T18:55:50.238 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:55:50.238 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-02-19T18:55:50.238 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-02-19T18:55:50.238 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph Dashboard is now available at: 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout: URL: https://smithi039.front.sepia.ceph.com:8443/ 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout: User: admin 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout: Password: r1neyd015n 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:50.240 INFO:teuthology.orchestra.run.smithi039.stdout:Saving cluster configuration to /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config directory 2024-02-19T18:55:50.241 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling autotune for osd_memory_target 2024-02-19T18:55:51.180 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:50 smithi039 ceph-mon[28988]: mgrmap e13: smithi039.ydmpuh(active, since 6s) 2024-02-19T18:55:51.446 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9be5c2a700 1 Processor -- start 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9be5c2a700 1 -- start start 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9be5c2a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be01092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9be5c2a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be0079e80 con 0x7f9be0106ee0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9bdf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be01092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9bdf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be01092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53180/0 (socket says 172.21.15.39:53180) 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9bdf7fe700 1 -- 172.21.15.39:0/3027155063 learned_addr learned my addr 172.21.15.39:0/3027155063 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9bdf7fe700 1 -- 172.21.15.39:0/3027155063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be0109810 con 0x7f9be0106ee0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.655+0000 7f9bdf7fe700 1 --2- 172.21.15.39:0/3027155063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be01092d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00098a0 tx=0x7f9bd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a3157095201f3c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9bde7fc700 1 -- 172.21.15.39:0/3027155063 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd0004180 con 0x7f9be0106ee0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9bde7fc700 1 -- 172.21.15.39:0/3027155063 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9bd00042e0 con 0x7f9be0106ee0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9bde7fc700 1 -- 172.21.15.39:0/3027155063 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd0011af0 con 0x7f9be0106ee0 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 -- 172.21.15.39:0/3027155063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 msgr2=0x7f9be01092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:51.447 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 --2- 172.21.15.39:0/3027155063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be01092d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00098a0 tx=0x7f9bd0009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 -- 172.21.15.39:0/3027155063 shutdown_connections 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 --2- 172.21.15.39:0/3027155063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be01092d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 -- 172.21.15.39:0/3027155063 >> 172.21.15.39:0/3027155063 conn(0x7f9be01008b0 msgr2=0x7f9be0102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 -- 172.21.15.39:0/3027155063 shutdown_connections 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.656+0000 7f9be5c2a700 1 -- 172.21.15.39:0/3027155063 wait complete. 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9be5c2a700 1 Processor -- start 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9be5c2a700 1 -- start start 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9be5c2a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be0106840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9be5c2a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be0106d80 con 0x7f9be0106ee0 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9bdf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9bdf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53196/0 (socket says 172.21.15.39:53196) 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.657+0000 7f9bdf7fe700 1 -- 172.21.15.39:0/432139137 learned_addr learned my addr 172.21.15.39:0/432139137 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9bdf7fe700 1 -- 172.21.15.39:0/432139137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0009550 con 0x7f9be0106ee0 2024-02-19T18:55:51.448 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9bdf7fe700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be0106840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0011670 tx=0x7f9bd0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd0003a40 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9bd0003ba0 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be0104e90 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd0024820 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.658+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be0105330 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.659+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f9bd0024ae0 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.659+0000 7f9bdcff9700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9bc80383d0 0x7f9bc803a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.659+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be010a5a0 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.659+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9bd004c550 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.659+0000 7f9bdeffd700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9bc80383d0 0x7f9bc803a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.659+0000 7f9bdeffd700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9bc80383d0 0x7f9bc803a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4006fd0 tx=0x7f9bd4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.664+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9bd0024de0 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.860+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f9be010a790 con 0x7f9be0106ee0 2024-02-19T18:55:51.449 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.860+0000 7f9bdcff9700 1 -- 172.21.15.39:0/432139137 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f9bd0018b40 con 0x7f9be0106ee0 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9bc80383d0 msgr2=0x7f9bc803a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9bc80383d0 0x7f9bc803a890 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4006fd0 tx=0x7f9bd4006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 msgr2=0x7f9be0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be0106840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0011670 tx=0x7f9bd0011aa0 comp rx=0 tx=0).stop 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 shutdown_connections 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9bc80383d0 0x7f9bc803a890 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 --2- 172.21.15.39:0/432139137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9be0106ee0 0x7f9be0106840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 >> 172.21.15.39:0/432139137 conn(0x7f9be01008b0 msgr2=0x7f9be0101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 shutdown_connections 2024-02-19T18:55:51.450 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:50.862+0000 7f9be5c2a700 1 -- 172.21.15.39:0/432139137 wait complete. 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f6860174700 1 Processor -- start 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f6860174700 1 -- start start 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f6860174700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f6858106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f6860174700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6858106e10 con 0x7f6858106420 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f685df10700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f6858106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f685df10700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f6858106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53210/0 (socket says 172.21.15.39:53210) 2024-02-19T18:55:52.788 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f685df10700 1 -- 172.21.15.39:0/627983152 learned_addr learned my addr 172.21.15.39:0/627983152 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.861+0000 7f685df10700 1 -- 172.21.15.39:0/627983152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6858107620 con 0x7f6858106420 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f685df10700 1 --2- 172.21.15.39:0/627983152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f6858106840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f68480098a0 tx=0x7f6848009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d810995cf3549bf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f685cf0e700 1 -- 172.21.15.39:0/627983152 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6848004180 con 0x7f6858106420 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f685cf0e700 1 -- 172.21.15.39:0/627983152 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f68480042e0 con 0x7f6858106420 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f685cf0e700 1 -- 172.21.15.39:0/627983152 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6848011af0 con 0x7f6858106420 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f6860174700 1 -- 172.21.15.39:0/627983152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 msgr2=0x7f6858106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f6860174700 1 --2- 172.21.15.39:0/627983152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f6858106840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f68480098a0 tx=0x7f6848009bb0 comp rx=0 tx=0).stop 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f6860174700 1 -- 172.21.15.39:0/627983152 shutdown_connections 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f6860174700 1 --2- 172.21.15.39:0/627983152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f6858106840 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.862+0000 7f6860174700 1 -- 172.21.15.39:0/627983152 >> 172.21.15.39:0/627983152 conn(0x7f6858101a00 msgr2=0x7f6858103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f6860174700 1 -- 172.21.15.39:0/627983152 shutdown_connections 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f6860174700 1 -- 172.21.15.39:0/627983152 wait complete. 2024-02-19T18:55:52.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f6860174700 1 Processor -- start 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f6860174700 1 -- start start 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f6860174700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f685807bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f6860174700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685807c1a0 con 0x7f6858106420 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f685df10700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f685807bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f685df10700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f685807bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53218/0 (socket says 172.21.15.39:53218) 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.863+0000 7f685df10700 1 -- 172.21.15.39:0/2361297065 learned_addr learned my addr 172.21.15.39:0/2361297065 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f685df10700 1 -- 172.21.15.39:0/2361297065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848009550 con 0x7f6858106420 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f685df10700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f685807bc60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6848000c00 tx=0x7f6848011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6848003a70 con 0x7f6858106420 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6848003bd0 con 0x7f6858106420 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685807a2b0 con 0x7f6858106420 2024-02-19T18:55:52.790 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68480238c0 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.864+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685807a750 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.865+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f6848023b80 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.865+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68580623c0 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.865+0000 7f684effd700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6844038410 0x7f684403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.865+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f684804c3c0 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.865+0000 7f685d70f700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6844038410 0x7f684403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.866+0000 7f685d70f700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6844038410 0x7f684403a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6854006fd0 tx=0x7f6854006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:51.870+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f684801a430 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.181+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f685807afa0 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.184+0000 7f684effd700 1 -- 172.21.15.39:0/2361297065 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f6848018b00 con 0x7f6858106420 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6844038410 msgr2=0x7f684403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:52.791 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6844038410 0x7f684403a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6854006fd0 tx=0x7f6854006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 msgr2=0x7f685807bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f685807bc60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6848000c00 tx=0x7f6848011b50 comp rx=0 tx=0).stop 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 shutdown_connections 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6844038410 0x7f684403a8d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 --2- 172.21.15.39:0/2361297065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6858106420 0x7f685807bc60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 >> 172.21.15.39:0/2361297065 conn(0x7f6858101a00 msgr2=0x7f68581027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 shutdown_connections 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-02-19T18:55:52.186+0000 7f6860174700 1 -- 172.21.15.39:0/2361297065 wait complete. 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:52.792 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout:Or, if you are only running a single cluster on this host: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: ceph telemetry on 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout:For more information see: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:55:52.793 INFO:teuthology.orchestra.run.smithi039.stdout:Bootstrap complete. 2024-02-19T18:55:52.820 INFO:tasks.cephadm:Fetching config... 2024-02-19T18:55:52.820 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:55:52.820 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T18:55:52.835 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T18:55:52.836 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:55:52.836 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T18:55:52.891 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T18:55:52.891 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:55:52.891 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/keyring of=/dev/stdout 2024-02-19T18:55:52.958 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T18:55:52.958 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:55:52.958 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T18:55:53.013 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T18:55:53.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm9U5NY9dpvY/KtGw3Vy0jmjxpe5V8EsBCPIdrBqCd8MkySdOXdXjEXFCIadsv/W/zY2QBtHxEQGL6Xi+1atEb2DhWr6yO5vh7MaM6kk4idHQm/wyXcmv0IyXSVhjv7XfcDmaXvNOSYgnToqjJ+KQqj1PlajydYwsfldl0N48r6hEgP0IGw4RU7SfZ6W+RAHL3haXiFUywSBi36HcthpImnpBrRu6P04QoKQmtnOJbqDtOjwostOty42mL1VdBiFcfJEEsIrLJzaYrgoIy7nf+a0uCVqo/Se3CLx2xkdSrscYYg7F8OTnrPxkSMHz578D/9nOrgXteZ0wkV9yrs2Kz5U7PIxVrHhRQ8sZx+zp1W5BVAQ26g9DVNKrbUrVETJYSyNkzGfEUngdyyWim+SPjOR3cfzxZWOQo3WIxfzu62ASHvs35t6gjINy+8EnSsUvE4fAMntxpyYAn67O+RED2yObPtZMDavOMlWvynJnVdd7ossx88KZldQ0CyeBl2s8= ceph-1026b4de-cf58-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T18:55:53.102 INFO:teuthology.orchestra.run.smithi039.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm9U5NY9dpvY/KtGw3Vy0jmjxpe5V8EsBCPIdrBqCd8MkySdOXdXjEXFCIadsv/W/zY2QBtHxEQGL6Xi+1atEb2DhWr6yO5vh7MaM6kk4idHQm/wyXcmv0IyXSVhjv7XfcDmaXvNOSYgnToqjJ+KQqj1PlajydYwsfldl0N48r6hEgP0IGw4RU7SfZ6W+RAHL3haXiFUywSBi36HcthpImnpBrRu6P04QoKQmtnOJbqDtOjwostOty42mL1VdBiFcfJEEsIrLJzaYrgoIy7nf+a0uCVqo/Se3CLx2xkdSrscYYg7F8OTnrPxkSMHz578D/9nOrgXteZ0wkV9yrs2Kz5U7PIxVrHhRQ8sZx+zp1W5BVAQ26g9DVNKrbUrVETJYSyNkzGfEUngdyyWim+SPjOR3cfzxZWOQo3WIxfzu62ASHvs35t6gjINy+8EnSsUvE4fAMntxpyYAn67O+RED2yObPtZMDavOMlWvynJnVdd7ossx88KZldQ0CyeBl2s8= ceph-1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:55:53.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm9U5NY9dpvY/KtGw3Vy0jmjxpe5V8EsBCPIdrBqCd8MkySdOXdXjEXFCIadsv/W/zY2QBtHxEQGL6Xi+1atEb2DhWr6yO5vh7MaM6kk4idHQm/wyXcmv0IyXSVhjv7XfcDmaXvNOSYgnToqjJ+KQqj1PlajydYwsfldl0N48r6hEgP0IGw4RU7SfZ6W+RAHL3haXiFUywSBi36HcthpImnpBrRu6P04QoKQmtnOJbqDtOjwostOty42mL1VdBiFcfJEEsIrLJzaYrgoIy7nf+a0uCVqo/Se3CLx2xkdSrscYYg7F8OTnrPxkSMHz578D/9nOrgXteZ0wkV9yrs2Kz5U7PIxVrHhRQ8sZx+zp1W5BVAQ26g9DVNKrbUrVETJYSyNkzGfEUngdyyWim+SPjOR3cfzxZWOQo3WIxfzu62ASHvs35t6gjINy+8EnSsUvE4fAMntxpyYAn67O+RED2yObPtZMDavOMlWvynJnVdd7ossx88KZldQ0CyeBl2s8= ceph-1026b4de-cf58-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T18:55:53.163 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm9U5NY9dpvY/KtGw3Vy0jmjxpe5V8EsBCPIdrBqCd8MkySdOXdXjEXFCIadsv/W/zY2QBtHxEQGL6Xi+1atEb2DhWr6yO5vh7MaM6kk4idHQm/wyXcmv0IyXSVhjv7XfcDmaXvNOSYgnToqjJ+KQqj1PlajydYwsfldl0N48r6hEgP0IGw4RU7SfZ6W+RAHL3haXiFUywSBi36HcthpImnpBrRu6P04QoKQmtnOJbqDtOjwostOty42mL1VdBiFcfJEEsIrLJzaYrgoIy7nf+a0uCVqo/Se3CLx2xkdSrscYYg7F8OTnrPxkSMHz578D/9nOrgXteZ0wkV9yrs2Kz5U7PIxVrHhRQ8sZx+zp1W5BVAQ26g9DVNKrbUrVETJYSyNkzGfEUngdyyWim+SPjOR3cfzxZWOQo3WIxfzu62ASHvs35t6gjINy+8EnSsUvE4fAMntxpyYAn67O+RED2yObPtZMDavOMlWvynJnVdd7ossx88KZldQ0CyeBl2s8= ceph-1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:55:53.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T18:55:53.380 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:55:53.493 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:53 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2361297065' entity='client.admin' 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 -- 172.21.15.39:0/805956712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 msgr2=0x7f34040ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 --2- 172.21.15.39:0/805956712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34040ffba0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3400009910 tx=0x7f3400009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 -- 172.21.15.39:0/805956712 shutdown_connections 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 --2- 172.21.15.39:0/805956712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34040ffba0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 -- 172.21.15.39:0/805956712 >> 172.21.15.39:0/805956712 conn(0x7f34040fad20 msgr2=0x7f34040fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 -- 172.21.15.39:0/805956712 shutdown_connections 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.712+0000 7f340b2b1700 1 -- 172.21.15.39:0/805956712 wait complete. 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340b2b1700 1 Processor -- start 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340b2b1700 1 -- start start 2024-02-19T18:55:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340b2b1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34041948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340b2b1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3404194de0 con 0x7f34040ff780 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340904d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34041948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340904d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34041948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53224/0 (socket says 172.21.15.39:53224) 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340904d700 1 -- 172.21.15.39:0/1100322668 learned_addr learned my addr 172.21.15.39:0/1100322668 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.713+0000 7f340904d700 1 -- 172.21.15.39:0/1100322668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34000095f0 con 0x7f34040ff780 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.714+0000 7f340904d700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34041948a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3400005b40 tx=0x7f3400004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.714+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f340001d070 con 0x7f34040ff780 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.714+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3404194fe0 con 0x7f34040ff780 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.714+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f340000be10 con 0x7f34040ff780 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.714+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f340000f460 con 0x7f34040ff780 2024-02-19T18:55:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.714+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3404195480 con 0x7f34040ff780 2024-02-19T18:55:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.715+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f34000227e0 con 0x7f34040ff780 2024-02-19T18:55:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.715+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340418e3a0 con 0x7f34040ff780 2024-02-19T18:55:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.715+0000 7f33fa7fc700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33f0038410 0x7f33f003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.715+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f340004c360 con 0x7f34040ff780 2024-02-19T18:55:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.715+0000 7f340884c700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33f0038410 0x7f33f003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.716+0000 7f340884c700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33f0038410 0x7f33f003a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f33f4006fd0 tx=0x7f33f4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.720+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f340002ba00 con 0x7f34040ff780 2024-02-19T18:55:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.925+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f340402cc70 con 0x7f34040ff780 2024-02-19T18:55:55.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.928+0000 7f33fa7fc700 1 -- 172.21.15.39:0/1100322668 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3400027030 con 0x7f34040ff780 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33f0038410 msgr2=0x7f33f003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33f0038410 0x7f33f003a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f33f4006fd0 tx=0x7f33f4006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 msgr2=0x7f34041948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34041948a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3400005b40 tx=0x7f3400004740 comp rx=0 tx=0).stop 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 shutdown_connections 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33f0038410 0x7f33f003a8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 --2- 172.21.15.39:0/1100322668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34040ff780 0x7f34041948a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 >> 172.21.15.39:0/1100322668 conn(0x7f34040fad20 msgr2=0x7f34040fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 shutdown_connections 2024-02-19T18:55:55.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:55:55.931+0000 7f340b2b1700 1 -- 172.21.15.39:0/1100322668 wait complete. 2024-02-19T18:55:56.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:56.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: Deploying daemon ceph-exporter.smithi039 on smithi039 2024-02-19T18:55:56.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:55:56 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1100322668' entity='client.admin' 2024-02-19T18:55:56.664 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-19T18:55:56.664 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-19T18:55:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 -- 172.21.15.39:0/3354781888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 msgr2=0x7f8bd80ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 --2- 172.21.15.39:0/3354781888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd80ffbb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009910 tx=0x7f8bc8009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 -- 172.21.15.39:0/3354781888 shutdown_connections 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 --2- 172.21.15.39:0/3354781888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd80ffbb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 -- 172.21.15.39:0/3354781888 >> 172.21.15.39:0/3354781888 conn(0x7f8bd80fad30 msgr2=0x7f8bd80fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 -- 172.21.15.39:0/3354781888 shutdown_connections 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.207+0000 7f8bdcff9700 1 -- 172.21.15.39:0/3354781888 wait complete. 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bdcff9700 1 Processor -- start 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bdcff9700 1 -- start start 2024-02-19T18:56:00.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bdcff9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd81959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bdcff9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd8195ef0 con 0x7f8bd80ff790 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bd659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd81959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bd659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd81959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42912/0 (socket says 172.21.15.39:42912) 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.208+0000 7f8bd659c700 1 -- 172.21.15.39:0/759952772 learned_addr learned my addr 172.21.15.39:0/759952772 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bd659c700 1 -- 172.21.15.39:0/759952772 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80095f0 con 0x7f8bd80ff790 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bd659c700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd81959b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8005b40 tx=0x7f8bc8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bc801d070 con 0x7f8bd80ff790 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8bc800be10 con 0x7f8bd80ff790 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd81960f0 con 0x7f8bd80ff790 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bc800f460 con 0x7f8bd80ff790 2024-02-19T18:56:00.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.209+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd8196590 con 0x7f8bd80ff790 2024-02-19T18:56:00.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.210+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f8bc800f6d0 con 0x7f8bd80ff790 2024-02-19T18:56:00.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.210+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd81055d0 con 0x7f8bd80ff790 2024-02-19T18:56:00.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.210+0000 7f8bc77fe700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f8bc00383c0 0x7f8bc003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:00.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.210+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8bc804c440 con 0x7f8bd80ff790 2024-02-19T18:56:00.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.210+0000 7f8bd5d9b700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f8bc00383c0 0x7f8bc003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:00.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.211+0000 7f8bd5d9b700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f8bc00383c0 0x7f8bc003a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc006fd0 tx=0x7f8bcc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:00.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.216+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8bc802a360 con 0x7f8bd80ff790 2024-02-19T18:56:00.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.440+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bd80fc650 con 0x7f8bc00383c0 2024-02-19T18:56:00.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.443+0000 7f8bc77fe700 1 -- 172.21.15.39:0/759952772 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8bd80fc650 con 0x7f8bc00383c0 2024-02-19T18:56:00.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f8bc00383c0 msgr2=0x7f8bc003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:00.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f8bc00383c0 0x7f8bc003a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc006fd0 tx=0x7f8bcc006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:00.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 msgr2=0x7f8bd81959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:00.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd81959b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8005b40 tx=0x7f8bc8004740 comp rx=0 tx=0).stop 2024-02-19T18:56:00.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 shutdown_connections 2024-02-19T18:56:00.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f8bc00383c0 0x7f8bc003a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:00.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 --2- 172.21.15.39:0/759952772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8bd80ff790 0x7f8bd81959b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:00.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 >> 172.21.15.39:0/759952772 conn(0x7f8bd80fad30 msgr2=0x7f8bd80fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:00.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 shutdown_connections 2024-02-19T18:56:00.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:00.445+0000 7f8bdcff9700 1 -- 172.21.15.39:0/759952772 wait complete. 2024-02-19T18:56:01.081 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: Deploying daemon crash.smithi039 on smithi039 2024-02-19T18:56:01.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:00 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:01.104 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-02-19T18:56:01.104 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:56:01.104 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-02-19T18:56:01.122 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:56:01.122 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:56:01.179 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-02-19T18:56:01.179 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch host add smithi086 2024-02-19T18:56:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:56:02.021 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:01 smithi039 ceph-mon[28988]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 -- 172.21.15.39:0/3020044109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 msgr2=0x7f54281011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 --2- 172.21.15.39:0/3020044109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281011c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5418009910 tx=0x7f5418009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 -- 172.21.15.39:0/3020044109 shutdown_connections 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 --2- 172.21.15.39:0/3020044109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281011c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 -- 172.21.15.39:0/3020044109 >> 172.21.15.39:0/3020044109 conn(0x7f54280fa490 msgr2=0x7f54280fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 -- 172.21.15.39:0/3020044109 shutdown_connections 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.864+0000 7f542e4eb700 1 -- 172.21.15.39:0/3020044109 wait complete. 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f542e4eb700 1 Processor -- start 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f542e4eb700 1 -- start start 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f542e4eb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f542e4eb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5428195f00 con 0x7f5428100da0 2024-02-19T18:56:03.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f5427fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f5427fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42936/0 (socket says 172.21.15.39:42936) 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.865+0000 7f5427fff700 1 -- 172.21.15.39:0/957095965 learned_addr learned my addr 172.21.15.39:0/957095965 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f5427fff700 1 -- 172.21.15.39:0/957095965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54180095f0 con 0x7f5428100da0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f5427fff700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281959c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5418005b40 tx=0x7f5418004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f541801d070 con 0x7f5428100da0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5428196100 con 0x7f5428100da0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f541800be10 con 0x7f5428100da0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f541800f460 con 0x7f5428100da0 2024-02-19T18:56:03.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.866+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54281965a0 con 0x7f5428100da0 2024-02-19T18:56:03.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.867+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f541800f5c0 con 0x7f5428100da0 2024-02-19T18:56:03.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.867+0000 7f54257fa700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f5410038410 0x7f541003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:03.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.867+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f541804d410 con 0x7f5428100da0 2024-02-19T18:56:03.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.867+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542818f3d0 con 0x7f5428100da0 2024-02-19T18:56:03.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.867+0000 7f54277fe700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f5410038410 0x7f541003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:03.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.868+0000 7f54277fe700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f5410038410 0x7f541003a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f541c006fd0 tx=0x7f541c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:03.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:03.876+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5418027070 con 0x7f5428100da0 2024-02-19T18:56:04.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:04.091+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7f54280611d0 con 0x7f5410038410 2024-02-19T18:56:04.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:04.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:04.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:04.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:04.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: Deploying daemon node-exporter.smithi039 on smithi039 2024-02-19T18:56:04.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:04.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:04 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:05.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:05 smithi039 ceph-mon[28988]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:05.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:05 smithi039 ceph-mon[28988]: Deploying cephadm binary to smithi086 2024-02-19T18:56:06.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.751+0000 7f54257fa700 1 -- 172.21.15.39:0/957095965 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f54280611d0 con 0x7f5410038410 2024-02-19T18:56:06.770 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f5410038410 msgr2=0x7f541003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f5410038410 0x7f541003a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f541c006fd0 tx=0x7f541c006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 msgr2=0x7f54281959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281959c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5418005b40 tx=0x7f5418004740 comp rx=0 tx=0).stop 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 shutdown_connections 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f5410038410 0x7f541003a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 --2- 172.21.15.39:0/957095965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428100da0 0x7f54281959c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 >> 172.21.15.39:0/957095965 conn(0x7f54280fa490 msgr2=0x7f54280ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 shutdown_connections 2024-02-19T18:56:06.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:06.753+0000 7f542e4eb700 1 -- 172.21.15.39:0/957095965 wait complete. 2024-02-19T18:56:06.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:06 smithi039 ceph-mon[28988]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:07.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T18:56:07.760 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:56:08.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:07 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:08.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:07 smithi039 ceph-mon[28988]: Added host smithi086 2024-02-19T18:56:08.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:08 smithi039 ceph-mon[28988]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:10.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 -- 172.21.15.39:0/172811706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 msgr2=0x7f221c100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:10.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 --2- 172.21.15.39:0/172811706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c100ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2210009910 tx=0x7f2210009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:10.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 -- 172.21.15.39:0/172811706 shutdown_connections 2024-02-19T18:56:10.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 --2- 172.21.15.39:0/172811706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c100ca0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 -- 172.21.15.39:0/172811706 >> 172.21.15.39:0/172811706 conn(0x7f221c0fbe60 msgr2=0x7f221c0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 -- 172.21.15.39:0/172811706 shutdown_connections 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.054+0000 7f2220a38700 1 -- 172.21.15.39:0/172811706 wait complete. 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.055+0000 7f2220a38700 1 Processor -- start 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.055+0000 7f2220a38700 1 -- start start 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.055+0000 7f2220a38700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.055+0000 7f2220a38700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c195ee0 con 0x7f221c100880 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f221a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f221a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c1959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34280/0 (socket says 172.21.15.39:34280) 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f221a59c700 1 -- 172.21.15.39:0/601279785 learned_addr learned my addr 172.21.15.39:0/601279785 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f221a59c700 1 -- 172.21.15.39:0/601279785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22100095f0 con 0x7f221c100880 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f221a59c700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c1959a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2210005b40 tx=0x7f2210004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221001d070 con 0x7f221c100880 2024-02-19T18:56:10.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f221000be10 con 0x7f221c100880 2024-02-19T18:56:10.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221c1960e0 con 0x7f221c100880 2024-02-19T18:56:10.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221000f460 con 0x7f221c100880 2024-02-19T18:56:10.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.056+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221c196580 con 0x7f221c100880 2024-02-19T18:56:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.057+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f221000f6d0 con 0x7f221c100880 2024-02-19T18:56:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.057+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221c18f4a0 con 0x7f221c100880 2024-02-19T18:56:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.057+0000 7f220b7fe700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f22040383c0 0x7f220403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.057+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f221004c3d0 con 0x7f221c100880 2024-02-19T18:56:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.057+0000 7f2219d9b700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f22040383c0 0x7f220403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:10.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.058+0000 7f2219d9b700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f22040383c0 0x7f220403a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f220c006fd0 tx=0x7f220c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:10.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.062+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f221002a360 con 0x7f221c100880 2024-02-19T18:56:10.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.267+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f221c0611d0 con 0x7f22040383c0 2024-02-19T18:56:10.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.268+0000 7f220b7fe700 1 -- 172.21.15.39:0/601279785 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f221c0611d0 con 0x7f22040383c0 2024-02-19T18:56:10.286 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T18:56:10.286 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-02-19T18:56:10.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f22040383c0 msgr2=0x7f220403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:10.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f22040383c0 0x7f220403a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f220c006fd0 tx=0x7f220c006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:10.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 msgr2=0x7f221c1959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c1959a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2210005b40 tx=0x7f2210004740 comp rx=0 tx=0).stop 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 shutdown_connections 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f22040383c0 0x7f220403a880 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 --2- 172.21.15.39:0/601279785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f221c100880 0x7f221c1959a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.270+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 >> 172.21.15.39:0/601279785 conn(0x7f221c0fbe60 msgr2=0x7f221c0fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.271+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 shutdown_connections 2024-02-19T18:56:10.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:10.271+0000 7f2220a38700 1 -- 172.21.15.39:0/601279785 wait complete. 2024-02-19T18:56:10.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:10 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:10.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:10 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:10.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:10 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:10.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:10 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:10.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:10 smithi039 ceph-mon[28988]: Deploying daemon alertmanager.smithi039 on smithi039 2024-02-19T18:56:10.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:10 smithi039 ceph-mon[28988]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:11.035 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T18:56:11.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T18:56:11.240 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:56:11.362 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:11 smithi039 ceph-mon[28988]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:56:12.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:12 smithi039 ceph-mon[28988]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.692+0000 7f33c5c96700 1 -- 172.21.15.39:0/2013788542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 msgr2=0x7f33c00ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.692+0000 7f33c5c96700 1 --2- 172.21.15.39:0/2013788542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c00ffba0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009910 tx=0x7f33b8009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 -- 172.21.15.39:0/2013788542 shutdown_connections 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 --2- 172.21.15.39:0/2013788542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c00ffba0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 -- 172.21.15.39:0/2013788542 >> 172.21.15.39:0/2013788542 conn(0x7f33c00fad00 msgr2=0x7f33c00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 -- 172.21.15.39:0/2013788542 shutdown_connections 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 -- 172.21.15.39:0/2013788542 wait complete. 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 Processor -- start 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 -- start start 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c01959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.693+0000 7f33c5c96700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c0195f10 con 0x7f33c00ff780 2024-02-19T18:56:13.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c01959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c01959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34286/0 (socket says 172.21.15.39:34286) 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bf7fe700 1 -- 172.21.15.39:0/1287127736 learned_addr learned my addr 172.21.15.39:0/1287127736 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bf7fe700 1 -- 172.21.15.39:0/1287127736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80095f0 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bf7fe700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c01959d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f33b8005b40 tx=0x7f33b8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33b801d070 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c0196110 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f33b800be10 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33b800f460 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.694+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c01965b0 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.695+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f33b800f6d0 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.695+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c018f3b0 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.695+0000 7f33bcff9700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33ac0383c0 0x7f33ac03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.695+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f33b804c590 con 0x7f33c00ff780 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.695+0000 7f33beffd700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33ac0383c0 0x7f33ac03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:13.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.696+0000 7f33beffd700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33ac0383c0 0x7f33ac03a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f33a8006fd0 tx=0x7f33a8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.700+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f33b802a360 con 0x7f33c00ff780 2024-02-19T18:56:13.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:13.910+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f33c002cc70 con 0x7f33c00ff780 2024-02-19T18:56:14.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.892+0000 7f33bcff9700 1 -- 172.21.15.39:0/1287127736 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f33b8027070 con 0x7f33c00ff780 2024-02-19T18:56:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.894+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33ac0383c0 msgr2=0x7f33ac03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.894+0000 7f33c5c96700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33ac0383c0 0x7f33ac03a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f33a8006fd0 tx=0x7f33a8006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.894+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 msgr2=0x7f33c01959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.894+0000 7f33c5c96700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c01959d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f33b8005b40 tx=0x7f33b8004740 comp rx=0 tx=0).stop 2024-02-19T18:56:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.895+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 shutdown_connections 2024-02-19T18:56:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.895+0000 7f33c5c96700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f33ac0383c0 0x7f33ac03a880 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.895+0000 7f33c5c96700 1 --2- 172.21.15.39:0/1287127736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33c00ff780 0x7f33c01959d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.895+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 >> 172.21.15.39:0/1287127736 conn(0x7f33c00fad00 msgr2=0x7f33c0069090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.895+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 shutdown_connections 2024-02-19T18:56:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:56:14.895+0000 7f33c5c96700 1 -- 172.21.15.39:0/1287127736 wait complete. 2024-02-19T18:56:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2024-02-19T18:56:14.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:14 smithi039 ceph-mon[28988]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:14.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:14 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:14.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:14 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1287127736' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T18:56:16.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:15 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1287127736' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T18:56:16.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:15 smithi039 ceph-mon[28988]: osdmap e4: 0 total, 0 up, 0 in 2024-02-19T18:56:16.637 INFO:tasks.cephadm:Adding mon.smithi039 on smithi039 2024-02-19T18:56:16.637 INFO:tasks.cephadm:Adding mon.smithi086 on smithi086 2024-02-19T18:56:16.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi039:172.21.15.39=smithi039;smithi086:172.21.15.86=smithi086' 2024-02-19T18:56:16.834 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:16.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:16 smithi039 ceph-mon[28988]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:18.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- 172.21.15.86:0/1530622521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 msgr2=0x7ff45c0ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 --2- 172.21.15.86:0/1530622521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c0ffbb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff450009910 tx=0x7ff450009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- 172.21.15.86:0/1530622521 shutdown_connections 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 --2- 172.21.15.86:0/1530622521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c0ffbb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- 172.21.15.86:0/1530622521 >> 172.21.15.86:0/1530622521 conn(0x7ff45c0fad30 msgr2=0x7ff45c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- 172.21.15.86:0/1530622521 shutdown_connections 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- 172.21.15.86:0/1530622521 wait complete. 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 Processor -- start 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- start start 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:18.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.609+0000 7ff462b80700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c198690 con 0x7ff45c0ff790 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff46091c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff46091c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:58684/0 (socket says 172.21.15.86:58684) 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff46091c700 1 -- 172.21.15.86:0/3399736752 learned_addr learned my addr 172.21.15.86:0/3399736752 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff46091c700 1 -- 172.21.15.86:0/3399736752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4500095f0 con 0x7ff45c0ff790 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff46091c700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c069440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff450005b40 tx=0x7ff450004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45001d070 con 0x7ff45c0ff790 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c1987d0 con 0x7ff45c0ff790 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff45000be10 con 0x7ff45c0ff790 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff45000f460 con 0x7ff45c0ff790 2024-02-19T18:56:18.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.610+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c198c70 con 0x7ff45c0ff790 2024-02-19T18:56:18.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.611+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c18e3b0 con 0x7ff45c0ff790 2024-02-19T18:56:18.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.611+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7ff4500227e0 con 0x7ff45c0ff790 2024-02-19T18:56:18.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.612+0000 7ff459ffb700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff44c0383c0 0x7ff44c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:18.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.612+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff45004c470 con 0x7ff45c0ff790 2024-02-19T18:56:18.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.612+0000 7ff45bfff700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff44c0383c0 0x7ff44c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:18.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.612+0000 7ff45bfff700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff44c0383c0 0x7ff44c03a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff448006fd0 tx=0x7ff448006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:18.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.617+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff45002bba0 con 0x7ff45c0ff790 2024-02-19T18:56:18.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.815+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 --> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi039:172.21.15.39=smithi039;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7ff45c061340 con 0x7ff44c0383c0 2024-02-19T18:56:18.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.822+0000 7ff459ffb700 1 -- 172.21.15.86:0/3399736752 <== mgr.14164 v2:172.21.15.39:6800/2158365942 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff45c061340 con 0x7ff44c0383c0 2024-02-19T18:56:18.839 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff44c0383c0 msgr2=0x7ff44c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff44c0383c0 0x7ff44c03a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff448006fd0 tx=0x7ff448006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 msgr2=0x7ff45c069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c069440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff450005b40 tx=0x7ff450004740 comp rx=0 tx=0).stop 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 shutdown_connections 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff44c0383c0 0x7ff44c03a880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 --2- 172.21.15.86:0/3399736752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff45c0ff790 0x7ff45c069440 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 >> 172.21.15.86:0/3399736752 conn(0x7ff45c0fad30 msgr2=0x7ff45c0fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 shutdown_connections 2024-02-19T18:56:18.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:18.824+0000 7ff462b80700 1 -- 172.21.15.86:0/3399736752 wait complete. 2024-02-19T18:56:19.180 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:18 smithi039 ceph-mon[28988]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:19.180 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:18 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:19.340 DEBUG:teuthology.orchestra.run.smithi086:mon.smithi086> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi086.service 2024-02-19T18:56:19.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:19.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:19.612 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:20.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:19 smithi039 ceph-mon[28988]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi039:172.21.15.39=smithi039;smithi086:172.21.15.86=smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:20.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:19 smithi039 ceph-mon[28988]: Saving service mon spec with placement smithi039:172.21.15.39=smithi039;smithi086:172.21.15.86=smithi086;count:2 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 -- 172.21.15.86:0/181872676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 msgr2=0x7f7498100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 --2- 172.21.15.86:0/181872676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498100ca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7488009910 tx=0x7f7488009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 -- 172.21.15.86:0/181872676 shutdown_connections 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 --2- 172.21.15.86:0/181872676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498100ca0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 -- 172.21.15.86:0/181872676 >> 172.21.15.86:0/181872676 conn(0x7f74980fbe60 msgr2=0x7f74980fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 -- 172.21.15.86:0/181872676 shutdown_connections 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 -- 172.21.15.86:0/181872676 wait complete. 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.050+0000 7f749cb94700 1 Processor -- start 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749cb94700 1 -- start start 2024-02-19T18:56:21.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749cb94700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749cb94700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7498195e70 con 0x7f7498100880 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:58706/0 (socket says 172.21.15.86:58706) 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749659c700 1 -- 172.21.15.86:0/2559473170 learned_addr learned my addr 172.21.15.86:0/2559473170 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749659c700 1 -- 172.21.15.86:0/2559473170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74880095f0 con 0x7f7498100880 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f749659c700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498195930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7488005b40 tx=0x7f7488004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.051+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f748801d070 con 0x7f7498100880 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.052+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7498196070 con 0x7f7498100880 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.052+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f748800be10 con 0x7f7498100880 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.052+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f748800f460 con 0x7f7498100880 2024-02-19T18:56:21.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.052+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7498196510 con 0x7f7498100880 2024-02-19T18:56:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.052+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f748800b960 con 0x7f7498100880 2024-02-19T18:56:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.052+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749818f4a0 con 0x7f7498100880 2024-02-19T18:56:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.053+0000 7f74877fe700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f74800383c0 0x7f748003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.053+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f748804c480 con 0x7f7498100880 2024-02-19T18:56:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.053+0000 7f7495d9b700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f74800383c0 0x7f748003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:21.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.053+0000 7f7495d9b700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f74800383c0 0x7f748003a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f748c006fd0 tx=0x7f748c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:21.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.058+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f748802a360 con 0x7f7498100880 2024-02-19T18:56:21.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:21.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:21.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:20 smithi039 ceph-mon[28988]: Deploying daemon grafana.smithi039 on smithi039 2024-02-19T18:56:21.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.320+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f749802cc70 con 0x7f7498100880 2024-02-19T18:56:21.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.321+0000 7f74877fe700 1 -- 172.21.15.86:0/2559473170 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f7488027030 con 0x7f7498100880 2024-02-19T18:56:21.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:21.338 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f74800383c0 msgr2=0x7f748003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f74800383c0 0x7f748003a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f748c006fd0 tx=0x7f748c006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 msgr2=0x7f7498195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498195930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7488005b40 tx=0x7f7488004740 comp rx=0 tx=0).stop 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 shutdown_connections 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f74800383c0 0x7f748003a880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 --2- 172.21.15.86:0/2559473170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100880 0x7f7498195930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 >> 172.21.15.86:0/2559473170 conn(0x7f74980fbe60 msgr2=0x7f74980fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 shutdown_connections 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:21.323+0000 7f749cb94700 1 -- 172.21.15.86:0/2559473170 wait complete. 2024-02-19T18:56:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:22.235 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:21 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/2559473170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:22.235 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:21 smithi039 ceph-mon[28988]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:22.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:22.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:23.061 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:24.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1111524372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 msgr2=0x7fd9040ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:24.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 --2- 172.21.15.86:0/1111524372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd9040ffba0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009910 tx=0x7fd8f4009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:24.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1111524372 shutdown_connections 2024-02-19T18:56:24.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 --2- 172.21.15.86:0/1111524372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd9040ffba0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:24.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1111524372 >> 172.21.15.86:0/1111524372 conn(0x7fd9040fad20 msgr2=0x7fd9040fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:24.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1111524372 shutdown_connections 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.571+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1111524372 wait complete. 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd90b3b9700 1 Processor -- start 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd90b3b9700 1 -- start start 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd90b3b9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd904068b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd90b3b9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd904198690 con 0x7fd9040ff780 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd909155700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd904068b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd909155700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd904068b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:48940/0 (socket says 172.21.15.86:48940) 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd909155700 1 -- 172.21.15.86:0/1404921307 learned_addr learned my addr 172.21.15.86:0/1404921307 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd909155700 1 -- 172.21.15.86:0/1404921307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f40095f0 con 0x7fd9040ff780 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.572+0000 7fd909155700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd904068b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4005b40 tx=0x7fd8f4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f401d070 con 0x7fd9040ff780 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9041987d0 con 0x7fd9040ff780 2024-02-19T18:56:24.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8f400be10 con 0x7fd9040ff780 2024-02-19T18:56:24.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f400f460 con 0x7fd9040ff780 2024-02-19T18:56:24.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904198c70 con 0x7fd9040ff780 2024-02-19T18:56:24.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fd8f400b960 con 0x7fd9040ff780 2024-02-19T18:56:24.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.573+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90418e3a0 con 0x7fd9040ff780 2024-02-19T18:56:24.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.574+0000 7fd8fa7fc700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd8f00383c0 0x7fd8f003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:24.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.574+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd8f404c3c0 con 0x7fd9040ff780 2024-02-19T18:56:24.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.574+0000 7fd908954700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd8f00383c0 0x7fd8f003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:24.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.574+0000 7fd908954700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd8f00383c0 0x7fd8f003a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:24.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.579+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd8f402a360 con 0x7fd9040ff780 2024-02-19T18:56:24.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.839+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd90402cc70 con 0x7fd9040ff780 2024-02-19T18:56:24.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.840+0000 7fd8fa7fc700 1 -- 172.21.15.86:0/1404921307 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fd8f4027030 con 0x7fd9040ff780 2024-02-19T18:56:24.857 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:24.857 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd8f00383c0 msgr2=0x7fd8f003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd8f00383c0 0x7fd8f003a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 msgr2=0x7fd904068b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd904068b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4005b40 tx=0x7fd8f4004740 comp rx=0 tx=0).stop 2024-02-19T18:56:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 shutdown_connections 2024-02-19T18:56:24.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fd8f00383c0 0x7fd8f003a880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:24.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 --2- 172.21.15.86:0/1404921307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9040ff780 0x7fd904068b50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:24.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 >> 172.21.15.86:0/1404921307 conn(0x7fd9040fad20 msgr2=0x7fd9040fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:24.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 shutdown_connections 2024-02-19T18:56:24.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:24.842+0000 7fd90b3b9700 1 -- 172.21.15.86:0/1404921307 wait complete. 2024-02-19T18:56:24.860 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:24.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:24 smithi039 ceph-mon[28988]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:24.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:24 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:26.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:25 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/1404921307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:26.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:26.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:27.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:26 smithi039 ceph-mon[28988]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 -- 172.21.15.86:0/1458196439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 msgr2=0x7f46dc0fef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 --2- 172.21.15.86:0/1458196439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc0fef70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009910 tx=0x7f46cc009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 -- 172.21.15.86:0/1458196439 shutdown_connections 2024-02-19T18:56:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 --2- 172.21.15.86:0/1458196439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc0fef70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 -- 172.21.15.86:0/1458196439 >> 172.21.15.86:0/1458196439 conn(0x7f46dc0fa490 msgr2=0x7f46dc0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 -- 172.21.15.86:0/1458196439 shutdown_connections 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.022+0000 7f46e1bab700 1 -- 172.21.15.86:0/1458196439 wait complete. 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46e1bab700 1 Processor -- start 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46e1bab700 1 -- start start 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46e1bab700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46e1bab700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc195f00 con 0x7f46dc0feb50 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46db7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46db7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:48952/0 (socket says 172.21.15.86:48952) 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.023+0000 7f46db7fe700 1 -- 172.21.15.86:0/3442490315 learned_addr learned my addr 172.21.15.86:0/3442490315 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46db7fe700 1 -- 172.21.15.86:0/3442490315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0095f0 con 0x7f46dc0feb50 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46db7fe700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc1959c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f46cc005950 tx=0x7f46cc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc01d070 con 0x7f46dc0feb50 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc196100 con 0x7f46dc0feb50 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc1965a0 con 0x7f46dc0feb50 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f46cc00be10 con 0x7f46dc0feb50 2024-02-19T18:56:28.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.024+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46cc00f460 con 0x7f46dc0feb50 2024-02-19T18:56:28.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.025+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f46cc00f5c0 con 0x7f46dc0feb50 2024-02-19T18:56:28.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.025+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc18f4c0 con 0x7f46dc0feb50 2024-02-19T18:56:28.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.025+0000 7f46d8ff9700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f46c40383c0 0x7f46c403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:28.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.025+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f46cc04c560 con 0x7f46dc0feb50 2024-02-19T18:56:28.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.025+0000 7f46daffd700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f46c40383c0 0x7f46c403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:28.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.026+0000 7f46daffd700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f46c40383c0 0x7f46c403a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46d0006fd0 tx=0x7f46d0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:28.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.030+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f46cc02a950 con 0x7f46dc0feb50 2024-02-19T18:56:28.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.291+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46dc0ff680 con 0x7f46dc0feb50 2024-02-19T18:56:28.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.292+0000 7f46d8ff9700 1 -- 172.21.15.86:0/3442490315 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f46dc0ff680 con 0x7f46dc0feb50 2024-02-19T18:56:28.309 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:28.309 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:28.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f46c40383c0 msgr2=0x7f46c403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:28.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f46c40383c0 0x7f46c403a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46d0006fd0 tx=0x7f46d0006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 msgr2=0x7f46dc1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc1959c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f46cc005950 tx=0x7f46cc009e60 comp rx=0 tx=0).stop 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 shutdown_connections 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f46c40383c0 0x7f46c403a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 --2- 172.21.15.86:0/3442490315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0feb50 0x7f46dc1959c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 >> 172.21.15.86:0/3442490315 conn(0x7f46dc0fa490 msgr2=0x7f46dc0fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 shutdown_connections 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:28.294+0000 7f46e1bab700 1 -- 172.21.15.86:0/3442490315 wait complete. 2024-02-19T18:56:28.312 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:29.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:28 smithi039 ceph-mon[28988]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:29.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:28 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3442490315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:30.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:30.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:30.283 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:31.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:30 smithi039 ceph-mon[28988]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:31.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.624+0000 7f0a60b20700 1 -- 172.21.15.86:0/2025731398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 msgr2=0x7f0a5c1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:31.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.624+0000 7f0a60b20700 1 --2- 172.21.15.86:0/2025731398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c1011c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c009910 tx=0x7f0a4c009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.624+0000 7f0a60b20700 1 -- 172.21.15.86:0/2025731398 shutdown_connections 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.624+0000 7f0a60b20700 1 --2- 172.21.15.86:0/2025731398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c1011c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 -- 172.21.15.86:0/2025731398 >> 172.21.15.86:0/2025731398 conn(0x7f0a5c0fa490 msgr2=0x7f0a5c0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 -- 172.21.15.86:0/2025731398 shutdown_connections 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 -- 172.21.15.86:0/2025731398 wait complete. 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 Processor -- start 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 -- start start 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.625+0000 7f0a60b20700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c1980f0 con 0x7f0a5c100da0 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a5a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a5a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:50992/0 (socket says 172.21.15.86:50992) 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a5a59c700 1 -- 172.21.15.86:0/2432568890 learned_addr learned my addr 172.21.15.86:0/2432568890 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a5a59c700 1 -- 172.21.15.86:0/2432568890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0095f0 con 0x7f0a5c100da0 2024-02-19T18:56:31.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a5a59c700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c197bb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005b40 tx=0x7f0a4c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:31.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a4c01d070 con 0x7f0a5c100da0 2024-02-19T18:56:31.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0a4c00be10 con 0x7f0a5c100da0 2024-02-19T18:56:31.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c1982f0 con 0x7f0a5c100da0 2024-02-19T18:56:31.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a4c00f460 con 0x7f0a5c100da0 2024-02-19T18:56:31.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.626+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c198790 con 0x7f0a5c100da0 2024-02-19T18:56:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.627+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f0a4c00f6d0 con 0x7f0a5c100da0 2024-02-19T18:56:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.627+0000 7f0a4b7fe700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f0a44038410 0x7f0a4403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.627+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c1917d0 con 0x7f0a5c100da0 2024-02-19T18:56:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.627+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0a4c04c490 con 0x7f0a5c100da0 2024-02-19T18:56:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.627+0000 7f0a59d9b700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f0a44038410 0x7f0a4403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:31.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.628+0000 7f0a59d9b700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f0a44038410 0x7f0a4403a8d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a50006fd0 tx=0x7f0a50006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:31.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.633+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0a4c02c440 con 0x7f0a5c100da0 2024-02-19T18:56:31.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.894+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a5c02cc70 con 0x7f0a5c100da0 2024-02-19T18:56:31.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.895+0000 7f0a4b7fe700 1 -- 172.21.15.86:0/2432568890 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0a4c027090 con 0x7f0a5c100da0 2024-02-19T18:56:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:31.912 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:31.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f0a44038410 msgr2=0x7f0a4403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:31.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f0a44038410 0x7f0a4403a8d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a50006fd0 tx=0x7f0a50006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 msgr2=0x7f0a5c197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c197bb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005b40 tx=0x7f0a4c004740 comp rx=0 tx=0).stop 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 shutdown_connections 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f0a44038410 0x7f0a4403a8d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 --2- 172.21.15.86:0/2432568890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a5c100da0 0x7f0a5c197bb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 >> 172.21.15.86:0/2432568890 conn(0x7f0a5c0fa490 msgr2=0x7f0a5c0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 shutdown_connections 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:31.897+0000 7f0a60b20700 1 -- 172.21.15.86:0/2432568890 wait complete. 2024-02-19T18:56:31.915 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:32.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:31 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/2432568890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:33.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:32 smithi039 ceph-mon[28988]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:33.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:33.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:33.728 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:34.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:34 smithi039 ceph-mon[28988]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/3697086001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 msgr2=0x7fe6e40ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 --2- 172.21.15.86:0/3697086001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40ffba0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400b0a0 tx=0x7fe6d400b3b0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/3697086001 shutdown_connections 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 --2- 172.21.15.86:0/3697086001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40ffba0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/3697086001 >> 172.21.15.86:0/3697086001 conn(0x7fe6e40fad20 msgr2=0x7fe6e40fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/3697086001 shutdown_connections 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.198+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/3697086001 wait complete. 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6ebd9e700 1 Processor -- start 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6ebd9e700 1 -- start start 2024-02-19T18:56:35.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6ebd9e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6ebd9e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40fd200 con 0x7fe6e40ff780 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e9b3a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e9b3a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40fccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:51012/0 (socket says 172.21.15.86:51012) 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e9b3a700 1 -- 172.21.15.86:0/4178854255 learned_addr learned my addr 172.21.15.86:0/4178854255 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e9b3a700 1 -- 172.21.15.86:0/4178854255 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d4009d00 con 0x7fe6e40ff780 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e9b3a700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40fccc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400b830 tx=0x7fe6d400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d400e040 con 0x7fe6e40ff780 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe6d4012af0 con 0x7fe6e40ff780 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.199+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e40fb310 con 0x7fe6e40ff780 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.200+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d401d9f0 con 0x7fe6e40ff780 2024-02-19T18:56:35.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.200+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e40fb730 con 0x7fe6e40ff780 2024-02-19T18:56:35.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.200+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fe6d401dcb0 con 0x7fe6e40ff780 2024-02-19T18:56:35.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.200+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e40fc140 con 0x7fe6e40ff780 2024-02-19T18:56:35.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.200+0000 7fe6e2ffd700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fe6cc0383c0 0x7fe6cc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:35.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.201+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe6d404c990 con 0x7fe6e40ff780 2024-02-19T18:56:35.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.201+0000 7fe6e9339700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fe6cc0383c0 0x7fe6cc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:35.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.201+0000 7fe6e9339700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fe6cc0383c0 0x7fe6cc03a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006fd0 tx=0x7fe6d8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:35.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.206+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe6d401ae20 con 0x7fe6e40ff780 2024-02-19T18:56:35.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.469+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6e418ed90 con 0x7fe6e40ff780 2024-02-19T18:56:35.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.470+0000 7fe6e2ffd700 1 -- 172.21.15.86:0/4178854255 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe6d401ac40 con 0x7fe6e40ff780 2024-02-19T18:56:35.488 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:35.488 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:35.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fe6cc0383c0 msgr2=0x7fe6cc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:35.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fe6cc0383c0 0x7fe6cc03a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006fd0 tx=0x7fe6d8006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:35.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 msgr2=0x7fe6e40fccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:35.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40fccc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400b830 tx=0x7fe6d400bd40 comp rx=0 tx=0).stop 2024-02-19T18:56:35.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 shutdown_connections 2024-02-19T18:56:35.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fe6cc0383c0 0x7fe6cc03a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 --2- 172.21.15.86:0/4178854255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6e40ff780 0x7fe6e40fccc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 >> 172.21.15.86:0/4178854255 conn(0x7fe6e40fad20 msgr2=0x7fe6e4068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:35.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 shutdown_connections 2024-02-19T18:56:35.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:35.472+0000 7fe6ebd9e700 1 -- 172.21.15.86:0/4178854255 wait complete. 2024-02-19T18:56:35.490 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:36.225 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:35 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/4178854255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:37.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:37.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:37.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:36 smithi039 ceph-mon[28988]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:37.251 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:38.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3252663563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 msgr2=0x7fbce8073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:38.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 --2- 172.21.15.86:0/3252663563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce8073220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009910 tx=0x7fbce4009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:38.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3252663563 shutdown_connections 2024-02-19T18:56:38.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 --2- 172.21.15.86:0/3252663563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce8073220 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:38.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3252663563 >> 172.21.15.86:0/3252663563 conn(0x7fbce80fbe20 msgr2=0x7fbce80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:38.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3252663563 shutdown_connections 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3252663563 wait complete. 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.676+0000 7fbcf02ae700 1 Processor -- start 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcf02ae700 1 -- start start 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcf02ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce819c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcf02ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce819c6a0 con 0x7fbce8074dc0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcee04a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce819c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcee04a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce819c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:51032/0 (socket says 172.21.15.86:51032) 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcee04a700 1 -- 172.21.15.86:0/3349353562 learned_addr learned my addr 172.21.15.86:0/3349353562 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcee04a700 1 -- 172.21.15.86:0/3349353562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce40095f0 con 0x7fbce8074dc0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcee04a700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce819c160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbce4005b40 tx=0x7fbce4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce401d070 con 0x7fbce8074dc0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbce400be10 con 0x7fbce8074dc0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce81044e0 con 0x7fbce8074dc0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce400f460 con 0x7fbce8074dc0 2024-02-19T18:56:38.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.677+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce819cbb0 con 0x7fbce8074dc0 2024-02-19T18:56:38.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.678+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fbce400b960 con 0x7fbce8074dc0 2024-02-19T18:56:38.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.678+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce8195c60 con 0x7fbce8074dc0 2024-02-19T18:56:38.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.678+0000 7fbcdf7fe700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fbcd40383c0 0x7fbcd403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:38.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.678+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbce404c480 con 0x7fbce8074dc0 2024-02-19T18:56:38.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.679+0000 7fbced849700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fbcd40383c0 0x7fbcd403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:38.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.679+0000 7fbced849700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fbcd40383c0 0x7fbcd403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8006fd0 tx=0x7fbcd8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:38.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.683+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbce402a360 con 0x7fbce8074dc0 2024-02-19T18:56:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.946+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbce80623c0 con 0x7fbce8074dc0 2024-02-19T18:56:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.946+0000 7fbcdf7fe700 1 -- 172.21.15.86:0/3349353562 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fbce4027030 con 0x7fbce8074dc0 2024-02-19T18:56:38.965 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:38.965 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fbcd40383c0 msgr2=0x7fbcd403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fbcd40383c0 0x7fbcd403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8006fd0 tx=0x7fbcd8006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 msgr2=0x7fbce819c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce819c160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbce4005b40 tx=0x7fbce4004740 comp rx=0 tx=0).stop 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 shutdown_connections 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fbcd40383c0 0x7fbcd403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 --2- 172.21.15.86:0/3349353562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce8074dc0 0x7fbce819c160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 >> 172.21.15.86:0/3349353562 conn(0x7fbce80fbe20 msgr2=0x7fbce80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:38.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 shutdown_connections 2024-02-19T18:56:38.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:38.948+0000 7fbcf02ae700 1 -- 172.21.15.86:0/3349353562 wait complete. 2024-02-19T18:56:38.968 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:39.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:39.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:39.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:38 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:40.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:39 smithi039 ceph-mon[28988]: Deploying daemon prometheus.smithi039 on smithi039 2024-02-19T18:56:40.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:39 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3349353562' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:40.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:40.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:40.733 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:40.778 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:41.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:40 smithi039 ceph-mon[28988]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:42.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 -- 172.21.15.86:0/1054600334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 msgr2=0x7ff900100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:42.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 --2- 172.21.15.86:0/1054600334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff900100cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009910 tx=0x7ff8f4009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:42.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 -- 172.21.15.86:0/1054600334 shutdown_connections 2024-02-19T18:56:42.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 --2- 172.21.15.86:0/1054600334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff900100cc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 -- 172.21.15.86:0/1054600334 >> 172.21.15.86:0/1054600334 conn(0x7ff9000fbe20 msgr2=0x7ff9000fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 -- 172.21.15.86:0/1054600334 shutdown_connections 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.070+0000 7ff906ba6700 1 -- 172.21.15.86:0/1054600334 wait complete. 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff906ba6700 1 Processor -- start 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff906ba6700 1 -- start start 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff906ba6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff9001959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff906ba6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff900195ee0 con 0x7ff9001008a0 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff904942700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff9001959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff904942700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff9001959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:40030/0 (socket says 172.21.15.86:40030) 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff904942700 1 -- 172.21.15.86:0/3394280755 learned_addr learned my addr 172.21.15.86:0/3394280755 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff904942700 1 -- 172.21.15.86:0/3394280755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40095f0 con 0x7ff9001008a0 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff904942700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff9001959a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4005b40 tx=0x7ff8f4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.071+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f401d070 con 0x7ff9001008a0 2024-02-19T18:56:42.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.072+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff8f400be10 con 0x7ff9001008a0 2024-02-19T18:56:42.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.072+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9001960e0 con 0x7ff9001008a0 2024-02-19T18:56:42.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.072+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8f400f460 con 0x7ff9001008a0 2024-02-19T18:56:42.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.072+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff900196580 con 0x7ff9001008a0 2024-02-19T18:56:42.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.073+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7ff8f400b960 con 0x7ff9001008a0 2024-02-19T18:56:42.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.073+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90018f4c0 con 0x7ff9001008a0 2024-02-19T18:56:42.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.073+0000 7ff8fdffb700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8f00383c0 0x7ff8f003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:42.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.073+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff8f404c4b0 con 0x7ff9001008a0 2024-02-19T18:56:42.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.073+0000 7ff8fffff700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8f00383c0 0x7ff8f003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:42.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.073+0000 7ff8fffff700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8f00383c0 0x7ff8f003a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec006fd0 tx=0x7ff8ec006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:42.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.078+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff8f402a360 con 0x7ff9001008a0 2024-02-19T18:56:42.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:41 smithi039 ceph-mon[28988]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:42.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.343+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff90002cc70 con 0x7ff9001008a0 2024-02-19T18:56:42.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.343+0000 7ff8fdffb700 1 -- 172.21.15.86:0/3394280755 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff8f4027030 con 0x7ff9001008a0 2024-02-19T18:56:42.361 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:42.361 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:42.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8f00383c0 msgr2=0x7ff8f003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8f00383c0 0x7ff8f003a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec006fd0 tx=0x7ff8ec006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 msgr2=0x7ff9001959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff9001959a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4005b40 tx=0x7ff8f4004740 comp rx=0 tx=0).stop 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 shutdown_connections 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7ff8f00383c0 0x7ff8f003a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 --2- 172.21.15.86:0/3394280755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9001008a0 0x7ff9001959a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 >> 172.21.15.86:0/3394280755 conn(0x7ff9000fbe20 msgr2=0x7ff9000fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 shutdown_connections 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:42.345+0000 7ff906ba6700 1 -- 172.21.15.86:0/3394280755 wait complete. 2024-02-19T18:56:42.363 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:43.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:42 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3394280755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:44.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:44.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:44.234 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:44.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:44 smithi039 ceph-mon[28988]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 -- 172.21.15.86:0/2107762135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 msgr2=0x7f6c20100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 --2- 172.21.15.86:0/2107762135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c20100cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009910 tx=0x7f6c14009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 -- 172.21.15.86:0/2107762135 shutdown_connections 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 --2- 172.21.15.86:0/2107762135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c20100cc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 -- 172.21.15.86:0/2107762135 >> 172.21.15.86:0/2107762135 conn(0x7f6c200fbe20 msgr2=0x7f6c200fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 -- 172.21.15.86:0/2107762135 shutdown_connections 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.776+0000 7f6c25902700 1 -- 172.21.15.86:0/2107762135 wait complete. 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c25902700 1 Processor -- start 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c25902700 1 -- start start 2024-02-19T18:56:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c25902700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c201959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c25902700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c20195ee0 con 0x7f6c201008a0 2024-02-19T18:56:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c1effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c201959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c1effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c201959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:40062/0 (socket says 172.21.15.86:40062) 2024-02-19T18:56:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.777+0000 7f6c1effd700 1 -- 172.21.15.86:0/3422088626 learned_addr learned my addr 172.21.15.86:0/3422088626 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c1effd700 1 -- 172.21.15.86:0/3422088626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140095f0 con 0x7f6c201008a0 2024-02-19T18:56:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c1effd700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c201959a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005b40 tx=0x7f6c14004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c1401d070 con 0x7f6c201008a0 2024-02-19T18:56:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c201960e0 con 0x7f6c201008a0 2024-02-19T18:56:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c1400be10 con 0x7f6c201008a0 2024-02-19T18:56:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c1400f460 con 0x7f6c201008a0 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.778+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c20196580 con 0x7f6c201008a0 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.779+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f6c1400b960 con 0x7f6c201008a0 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.779+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2018f4c0 con 0x7f6c201008a0 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.779+0000 7f6c24900700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6c08038410 0x7f6c0803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.779+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6c1404c420 con 0x7f6c201008a0 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.779+0000 7f6c1e7fc700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6c08038410 0x7f6c0803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.780+0000 7f6c1e7fc700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6c08038410 0x7f6c0803a8d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c10006fd0 tx=0x7f6c10006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:45.784+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6c1402bb60 con 0x7f6c201008a0 2024-02-19T18:56:46.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.044+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c2002cc70 con 0x7f6c201008a0 2024-02-19T18:56:46.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.045+0000 7f6c24900700 1 -- 172.21.15.86:0/3422088626 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6c14027030 con 0x7f6c201008a0 2024-02-19T18:56:46.062 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:46.062 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:46.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6c08038410 msgr2=0x7f6c0803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:46.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6c08038410 0x7f6c0803a8d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c10006fd0 tx=0x7f6c10006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:46.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 msgr2=0x7f6c201959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:46.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c201959a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005b40 tx=0x7f6c14004740 comp rx=0 tx=0).stop 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 shutdown_connections 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f6c08038410 0x7f6c0803a8d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 --2- 172.21.15.86:0/3422088626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c201008a0 0x7f6c201959a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 >> 172.21.15.86:0/3422088626 conn(0x7f6c200fbe20 msgr2=0x7f6c200fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 shutdown_connections 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:46.047+0000 7f6c25902700 1 -- 172.21.15.86:0/3422088626 wait complete. 2024-02-19T18:56:46.065 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:47.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:46 smithi039 ceph-mon[28988]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:47.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:46 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3422088626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:47.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:47.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:47.852 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:47.898 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:49.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:48 smithi039 ceph-mon[28988]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 -- 172.21.15.86:0/1701578993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 msgr2=0x7f633c0ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 --2- 172.21.15.86:0/1701578993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c0ffbb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6338009910 tx=0x7f6338009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 -- 172.21.15.86:0/1701578993 shutdown_connections 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 --2- 172.21.15.86:0/1701578993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c0ffbb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 -- 172.21.15.86:0/1701578993 >> 172.21.15.86:0/1701578993 conn(0x7f633c0fad30 msgr2=0x7f633c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 -- 172.21.15.86:0/1701578993 shutdown_connections 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 -- 172.21.15.86:0/1701578993 wait complete. 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.272+0000 7f6342efa700 1 Processor -- start 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6342efa700 1 -- start start 2024-02-19T18:56:49.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6342efa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c1948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6342efa700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c194de0 con 0x7f633c0ff790 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6340c96700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c1948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6340c96700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c1948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:40080/0 (socket says 172.21.15.86:40080) 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6340c96700 1 -- 172.21.15.86:0/1433640065 learned_addr learned my addr 172.21.15.86:0/1433640065 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6340c96700 1 -- 172.21.15.86:0/1433640065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63380095f0 con 0x7f633c0ff790 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6340c96700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c1948a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6338005b40 tx=0x7f6338004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f633801d070 con 0x7f633c0ff790 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f633800be10 con 0x7f633c0ff790 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f633800f460 con 0x7f633c0ff790 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.273+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c194fe0 con 0x7f633c0ff790 2024-02-19T18:56:49.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.274+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c195480 con 0x7f633c0ff790 2024-02-19T18:56:49.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.275+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f63380227e0 con 0x7f633c0ff790 2024-02-19T18:56:49.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.275+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c18e3b0 con 0x7f633c0ff790 2024-02-19T18:56:49.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.275+0000 7f6331ffb700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f63240383c0 0x7f632403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:49.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.275+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f633804c3b0 con 0x7f633c0ff790 2024-02-19T18:56:49.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.275+0000 7f6333fff700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f63240383c0 0x7f632403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:49.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.276+0000 7f6333fff700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f63240383c0 0x7f632403a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f632c006fd0 tx=0x7f632c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:49.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.280+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f633802bba0 con 0x7f633c0ff790 2024-02-19T18:56:49.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.544+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f633c02cc70 con 0x7f633c0ff790 2024-02-19T18:56:49.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.545+0000 7f6331ffb700 1 -- 172.21.15.86:0/1433640065 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6338027020 con 0x7f633c0ff790 2024-02-19T18:56:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:49.562 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:49.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f63240383c0 msgr2=0x7f632403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:49.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f63240383c0 0x7f632403a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f632c006fd0 tx=0x7f632c006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:49.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 msgr2=0x7f633c1948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c1948a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6338005b40 tx=0x7f6338004740 comp rx=0 tx=0).stop 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 shutdown_connections 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f63240383c0 0x7f632403a880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 --2- 172.21.15.86:0/1433640065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f633c0ff790 0x7f633c1948a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 >> 172.21.15.86:0/1433640065 conn(0x7f633c0fad30 msgr2=0x7f633c0fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 shutdown_connections 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:49.547+0000 7f6342efa700 1 -- 172.21.15.86:0/1433640065 wait complete. 2024-02-19T18:56:49.565 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:50.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:49 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/1433640065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:51.112 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:50 smithi039 ceph-mon[28988]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:51.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:51.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:51.459 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:51.505 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:52.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 -- 172.21.15.86:0/879743728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 msgr2=0x7f1f3c101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:52.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 --2- 172.21.15.86:0/879743728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c101140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00b0a0 tx=0x7f1f2c00b3b0 comp rx=0 tx=0).stop 2024-02-19T18:56:52.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 -- 172.21.15.86:0/879743728 shutdown_connections 2024-02-19T18:56:52.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 --2- 172.21.15.86:0/879743728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c101140 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 -- 172.21.15.86:0/879743728 >> 172.21.15.86:0/879743728 conn(0x7f1f3c0fa490 msgr2=0x7f1f3c0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 -- 172.21.15.86:0/879743728 shutdown_connections 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.890+0000 7f1f41bf9700 1 -- 172.21.15.86:0/879743728 wait complete. 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f41bf9700 1 Processor -- start 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f41bf9700 1 -- start start 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f41bf9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c0ff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f41bf9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1986c0 con 0x7f1f3c100d20 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f3b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c0ff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f3b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c0ff4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:36922/0 (socket says 172.21.15.86:36922) 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f3b7fe700 1 -- 172.21.15.86:0/3289318644 learned_addr learned my addr 172.21.15.86:0/3289318644 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.891+0000 7f1f3b7fe700 1 -- 172.21.15.86:0/3289318644 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c009d00 con 0x7f1f3c100d20 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.892+0000 7f1f3b7fe700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c0ff4d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009fd0 tx=0x7f1f2c00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.892+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f2c00e040 con 0x7f1f3c100d20 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.892+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f2c00bdc0 con 0x7f1f3c100d20 2024-02-19T18:56:52.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.892+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c198800 con 0x7f1f3c100d20 2024-02-19T18:56:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.892+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f2c01da40 con 0x7f1f3c100d20 2024-02-19T18:56:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.892+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c198c20 con 0x7f1f3c100d20 2024-02-19T18:56:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.893+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f1f2c01dc30 con 0x7f1f3c100d20 2024-02-19T18:56:52.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.893+0000 7f1f38ff9700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f1f24038460 0x7f1f2403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.893+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1f2c04c890 con 0x7f1f3c100d20 2024-02-19T18:56:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.893+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c18e3b0 con 0x7f1f3c100d20 2024-02-19T18:56:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.893+0000 7f1f3affd700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f1f24038460 0x7f1f2403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.893+0000 7f1f3affd700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f1f24038460 0x7f1f2403a920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006fd0 tx=0x7f1f30006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:52.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:52.898+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1f2c01c3e0 con 0x7f1f3c100d20 2024-02-19T18:56:53.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.167+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f3c02cc70 con 0x7f1f3c100d20 2024-02-19T18:56:53.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.168+0000 7f1f38ff9700 1 -- 172.21.15.86:0/3289318644 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1f2c02c200 con 0x7f1f3c100d20 2024-02-19T18:56:53.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:53.185 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f1f24038460 msgr2=0x7f1f2403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f1f24038460 0x7f1f2403a920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006fd0 tx=0x7f1f30006e40 comp rx=0 tx=0).stop 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 msgr2=0x7f1f3c0ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c0ff4d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009fd0 tx=0x7f1f2c00bb10 comp rx=0 tx=0).stop 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 shutdown_connections 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f1f24038460 0x7f1f2403a920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:53.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 --2- 172.21.15.86:0/3289318644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f3c100d20 0x7f1f3c0ff4d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:53.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 >> 172.21.15.86:0/3289318644 conn(0x7f1f3c0fa490 msgr2=0x7f1f3c0fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:53.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 shutdown_connections 2024-02-19T18:56:53.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:53.170+0000 7f1f41bf9700 1 -- 172.21.15.86:0/3289318644 wait complete. 2024-02-19T18:56:53.188 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:53.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:52 smithi039 ceph-mon[28988]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:53.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:52 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:53.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:52 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:53.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:52 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:53.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:52 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T18:56:54.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:53 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3289318644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:54.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:53 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:56:54.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:53 smithi039 ceph-mon[28988]: from='mgr.14164 172.21.15.39:0/1240726449' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T18:56:54.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:53 smithi039 ceph-mon[28988]: mgrmap e14: smithi039.ydmpuh(active, since 70s) 2024-02-19T18:56:54.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:54.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:55.052 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:55.097 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 -- 172.21.15.86:0/3996757857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 msgr2=0x7f9b6c0ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 --2- 172.21.15.86:0/3996757857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c0ffba0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009910 tx=0x7f9b5c009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 -- 172.21.15.86:0/3996757857 shutdown_connections 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 --2- 172.21.15.86:0/3996757857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c0ffba0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 -- 172.21.15.86:0/3996757857 >> 172.21.15.86:0/3996757857 conn(0x7f9b6c0fad00 msgr2=0x7f9b6c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 -- 172.21.15.86:0/3996757857 shutdown_connections 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.520+0000 7f9b72157700 1 -- 172.21.15.86:0/3996757857 wait complete. 2024-02-19T18:56:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b72157700 1 Processor -- start 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b72157700 1 -- start start 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b72157700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c1959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b72157700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c195ef0 con 0x7f9b6c0ff780 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b6b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c1959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b6b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c1959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:36952/0 (socket says 172.21.15.86:36952) 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.521+0000 7f9b6b7fe700 1 -- 172.21.15.86:0/1572136629 learned_addr learned my addr 172.21.15.86:0/1572136629 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b6b7fe700 1 -- 172.21.15.86:0/1572136629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c0095f0 con 0x7f9b6c0ff780 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b6b7fe700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c1959b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005950 tx=0x7f9b5c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b5c01d070 con 0x7f9b6c0ff780 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9b5c00be10 con 0x7f9b6c0ff780 2024-02-19T18:56:56.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c1960f0 con 0x7f9b6c0ff780 2024-02-19T18:56:56.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b5c00f460 con 0x7f9b6c0ff780 2024-02-19T18:56:56.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.522+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c196590 con 0x7f9b6c0ff780 2024-02-19T18:56:56.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.523+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7f9b5c00f690 con 0x7f9b6c0ff780 2024-02-19T18:56:56.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.523+0000 7f9b68ff9700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 0x7f9b5403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:56.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.523+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c18f3b0 con 0x7f9b6c0ff780 2024-02-19T18:56:56.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.523+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9b5c04c6d0 con 0x7f9b6c0ff780 2024-02-19T18:56:56.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.523+0000 7f9b6affd700 1 -- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 msgr2=0x7f9b5403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2158365942 2024-02-19T18:56:56.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.523+0000 7f9b6affd700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 0x7f9b5403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:56:56.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.528+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9b5c02a360 con 0x7f9b6c0ff780 2024-02-19T18:56:56.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.724+0000 7f9b6affd700 1 -- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 msgr2=0x7f9b5403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2158365942 2024-02-19T18:56:56.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.724+0000 7f9b6affd700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 0x7f9b5403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:56:56.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.793+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b6c02cc70 con 0x7f9b6c0ff780 2024-02-19T18:56:56.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.794+0000 7f9b68ff9700 1 -- 172.21.15.86:0/1572136629 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9b5c027070 con 0x7f9b6c0ff780 2024-02-19T18:56:56.811 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:56:56.811 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:56:56.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 msgr2=0x7f9b5403a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:56:56.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 0x7f9b5403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 msgr2=0x7f9b6c1959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c1959b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005950 tx=0x7f9b5c009e60 comp rx=0 tx=0).stop 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 shutdown_connections 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7f9b54038410 0x7f9b5403a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 --2- 172.21.15.86:0/1572136629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9b6c0ff780 0x7f9b6c1959b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 >> 172.21.15.86:0/1572136629 conn(0x7f9b6c0fad00 msgr2=0x7f9b6c0690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 shutdown_connections 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:56.796+0000 7f9b72157700 1 -- 172.21.15.86:0/1572136629 wait complete. 2024-02-19T18:56:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:56:57.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:56:56 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/1572136629' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:56:58.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:56:58.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:56:58.556 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:58.602 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 -- 172.21.15.86:0/370519949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 msgr2=0x7fae04100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 --2- 172.21.15.86:0/370519949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04100cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009910 tx=0x7fadf4009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 -- 172.21.15.86:0/370519949 shutdown_connections 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 --2- 172.21.15.86:0/370519949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04100cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 -- 172.21.15.86:0/370519949 >> 172.21.15.86:0/370519949 conn(0x7fae040fbe20 msgr2=0x7fae040fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 -- 172.21.15.86:0/370519949 shutdown_connections 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 -- 172.21.15.86:0/370519949 wait complete. 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 Processor -- start 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.972+0000 7fae0c2cd700 1 -- start start 2024-02-19T18:56:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0c2cd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0c2cd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae04075070 con 0x7fae041008a0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0a069700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0a069700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:36958/0 (socket says 172.21.15.86:36958) 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0a069700 1 -- 172.21.15.86:0/4053974670 learned_addr learned my addr 172.21.15.86:0/4053974670 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0a069700 1 -- 172.21.15.86:0/4053974670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40095f0 con 0x7fae041008a0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0a069700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04074b30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005950 tx=0x7fadf4009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf401d070 con 0x7fae041008a0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae04073180 con 0x7fae041008a0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fadf400be10 con 0x7fae041008a0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadf400f460 con 0x7fae041008a0 2024-02-19T18:56:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.973+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae04073620 con 0x7fae041008a0 2024-02-19T18:56:59.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.974+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040623c0 con 0x7fae041008a0 2024-02-19T18:56:59.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.974+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7fadf40227c0 con 0x7fae041008a0 2024-02-19T18:56:59.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.974+0000 7fadfb7fe700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 0x7fadf003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:59.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.974+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fadf404c530 con 0x7fae041008a0 2024-02-19T18:56:59.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.975+0000 7fae09868700 1 -- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 msgr2=0x7fadf003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2158365942 2024-02-19T18:56:59.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.975+0000 7fae09868700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 0x7fadf003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:56:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:56:59.979+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fadf402bba0 con 0x7fae041008a0 2024-02-19T18:57:00.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.176+0000 7fae09868700 1 -- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 msgr2=0x7fadf003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/2158365942 2024-02-19T18:57:00.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.176+0000 7fae09868700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 0x7fadf003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:57:00.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.240+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae0418eea0 con 0x7fae041008a0 2024-02-19T18:57:00.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.241+0000 7fadfb7fe700 1 -- 172.21.15.86:0/4053974670 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fadf4027030 con 0x7fae041008a0 2024-02-19T18:57:00.258 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:00.258 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.242+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 msgr2=0x7fadf003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.242+0000 7fae0c2cd700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 0x7fadf003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.242+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 msgr2=0x7fae04074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.242+0000 7fae0c2cd700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04074b30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005950 tx=0x7fadf4009e60 comp rx=0 tx=0).stop 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.243+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 shutdown_connections 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.243+0000 7fae0c2cd700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:6800/2158365942,v1:172.21.15.39:6801/2158365942] conn(0x7fadf0038410 0x7fadf003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.243+0000 7fae0c2cd700 1 --2- 172.21.15.86:0/4053974670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae041008a0 0x7fae04074b30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.243+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 >> 172.21.15.86:0/4053974670 conn(0x7fae040fbe20 msgr2=0x7fae040fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.243+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 shutdown_connections 2024-02-19T18:57:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:00.243+0000 7fae0c2cd700 1 -- 172.21.15.86:0/4053974670 wait complete. 2024-02-19T18:57:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:00.649 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:00 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/4053974670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:01.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: Active manager daemon smithi039.ydmpuh restarted 2024-02-19T18:57:01.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: Activating manager daemon smithi039.ydmpuh 2024-02-19T18:57:01.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:57:01.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: mgrmap e15: smithi039.ydmpuh(active, starting, since 0.00497014s) 2024-02-19T18:57:01.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:57:01.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:57:01.583 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T18:57:01.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:01.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:02.169 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:57:02.215 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:57:02.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: [19/Feb/2024:18:57:01] ENGINE Bus STARTING 2024-02-19T18:57:02.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: [19/Feb/2024:18:57:01] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T18:57:02.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: [19/Feb/2024:18:57:01] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T18:57:02.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: [19/Feb/2024:18:57:01] ENGINE Bus STARTED 2024-02-19T18:57:02.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:02.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: mgrmap e16: smithi039.ydmpuh(active, since 1.00703s) 2024-02-19T18:57:02.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:02 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 -- 172.21.15.86:0/2778855372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 msgr2=0x7f8f940ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 --2- 172.21.15.86:0/2778855372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f940ffba0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009910 tx=0x7f8f8c009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 -- 172.21.15.86:0/2778855372 shutdown_connections 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 --2- 172.21.15.86:0/2778855372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f940ffba0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 -- 172.21.15.86:0/2778855372 >> 172.21.15.86:0/2778855372 conn(0x7f8f940fad20 msgr2=0x7f8f940fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 -- 172.21.15.86:0/2778855372 shutdown_connections 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.570+0000 7f8f9b88a700 1 -- 172.21.15.86:0/2778855372 wait complete. 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f9b88a700 1 Processor -- start 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f9b88a700 1 -- start start 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f9b88a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f94197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f9b88a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f941980e0 con 0x7f8f940ff780 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f99626700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f94197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f99626700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f94197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:47534/0 (socket says 172.21.15.86:47534) 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.571+0000 7f8f99626700 1 -- 172.21.15.86:0/3246570923 learned_addr learned my addr 172.21.15.86:0/3246570923 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f99626700 1 -- 172.21.15.86:0/3246570923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c0095f0 con 0x7f8f940ff780 2024-02-19T18:57:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f99626700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f94197ba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c005b40 tx=0x7f8f8c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8c01d070 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f941982e0 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94198780 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f8c00be10 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.572+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8c00f460 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.573+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f8f8c00f5c0 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.573+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78005320 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.573+0000 7f8f8a7fc700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8f800384b0 0x7f8f8003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.573+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8f8c04c6e0 con 0x7f8f940ff780 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.573+0000 7f8f98e25700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8f800384b0 0x7f8f8003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.574+0000 7f8f98e25700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8f800384b0 0x7f8f8003a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8f84006fd0 tx=0x7f8f84006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.578+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8f8c00fa20 con 0x7f8f940ff780 2024-02-19T18:57:03.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.857+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f78005190 con 0x7f8f940ff780 2024-02-19T18:57:03.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.858+0000 7f8f8a7fc700 1 -- 172.21.15.86:0/3246570923 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8f8c027020 con 0x7f8f940ff780 2024-02-19T18:57:03.875 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:03.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8f800384b0 msgr2=0x7f8f8003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:03.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8f800384b0 0x7f8f8003a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8f84006fd0 tx=0x7f8f84006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:03.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 msgr2=0x7f8f94197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f94197ba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c005b40 tx=0x7f8f8c004740 comp rx=0 tx=0).stop 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 shutdown_connections 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8f800384b0 0x7f8f8003a970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 --2- 172.21.15.86:0/3246570923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f940ff780 0x7f8f94197ba0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 >> 172.21.15.86:0/3246570923 conn(0x7f8f940fad20 msgr2=0x7f8f94068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 shutdown_connections 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:03.860+0000 7f8f9b88a700 1 -- 172.21.15.86:0/3246570923 wait complete. 2024-02-19T18:57:03.878 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:03.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:04.895 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:04 smithi039 ceph-mon[28988]: mgrmap e17: smithi039.ydmpuh(active, since 2s) 2024-02-19T18:57:04.895 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:04 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3246570923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:06.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:06.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:06.588 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:57:06.634 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:57:07.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:07 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:07.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:07 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:07.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:07 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:07.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:07 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:07.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:07 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:08.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 -- 172.21.15.86:0/2796711995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 msgr2=0x7fe1700ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 --2- 172.21.15.86:0/2796711995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700ffba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe164009910 tx=0x7fe164009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 -- 172.21.15.86:0/2796711995 shutdown_connections 2024-02-19T18:57:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 --2- 172.21.15.86:0/2796711995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700ffba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 -- 172.21.15.86:0/2796711995 >> 172.21.15.86:0/2796711995 conn(0x7fe1700fad20 msgr2=0x7fe1700fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 -- 172.21.15.86:0/2796711995 shutdown_connections 2024-02-19T18:57:08.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.044+0000 7fe174e54700 1 -- 172.21.15.86:0/2796711995 wait complete. 2024-02-19T18:57:08.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe174e54700 1 Processor -- start 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe174e54700 1 -- start start 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe174e54700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe174e54700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700fd200 con 0x7fe1700ff780 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe16e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe16e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700fccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:47552/0 (socket says 172.21.15.86:47552) 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe16e59c700 1 -- 172.21.15.86:0/3053974853 learned_addr learned my addr 172.21.15.86:0/3053974853 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.045+0000 7fe16e59c700 1 -- 172.21.15.86:0/3053974853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1640095f0 con 0x7fe1700ff780 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.046+0000 7fe16e59c700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700fccc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe164005b40 tx=0x7fe164004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.046+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe16401d070 con 0x7fe1700ff780 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.046+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1700fb310 con 0x7fe1700ff780 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.046+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe16400be10 con 0x7fe1700ff780 2024-02-19T18:57:08.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.046+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe16400f460 con 0x7fe1700ff780 2024-02-19T18:57:08.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.046+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1700fb7b0 con 0x7fe1700ff780 2024-02-19T18:57:08.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.047+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fe16400f6d0 con 0x7fe1700ff780 2024-02-19T18:57:08.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.047+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1700623c0 con 0x7fe1700ff780 2024-02-19T18:57:08.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.047+0000 7fe15f7fe700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe1580384b0 0x7fe15803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:08.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.047+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fe16404c840 con 0x7fe1700ff780 2024-02-19T18:57:08.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.047+0000 7fe16dd9b700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe1580384b0 0x7fe15803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:08.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.048+0000 7fe16dd9b700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe1580384b0 0x7fe15803a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe160006fd0 tx=0x7fe160006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:08.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.052+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe16402a360 con 0x7fe1700ff780 2024-02-19T18:57:08.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.317+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe17018ed90 con 0x7fe1700ff780 2024-02-19T18:57:08.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.318+0000 7fe15f7fe700 1 -- 172.21.15.86:0/3053974853 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fe164027030 con 0x7fe1700ff780 2024-02-19T18:57:08.336 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:08.336 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe1580384b0 msgr2=0x7fe15803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe1580384b0 0x7fe15803a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe160006fd0 tx=0x7fe160006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 msgr2=0x7fe1700fccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700fccc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe164005b40 tx=0x7fe164004740 comp rx=0 tx=0).stop 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 shutdown_connections 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe1580384b0 0x7fe15803a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 --2- 172.21.15.86:0/3053974853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1700ff780 0x7fe1700fccc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 >> 172.21.15.86:0/3053974853 conn(0x7fe1700fad20 msgr2=0x7fe1700690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 shutdown_connections 2024-02-19T18:57:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:08.320+0000 7fe174e54700 1 -- 172.21.15.86:0/3053974853 wait complete. 2024-02-19T18:57:08.339 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:08.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:08 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3053974853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:09.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:09.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:10.057 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:57:10.104 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:57:11.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:11.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:11.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:11.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:11.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:11.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:11.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:11.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 -- 172.21.15.86:0/960932022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 msgr2=0x7faa90105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 --2- 172.21.15.86:0/960932022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa90105ae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009910 tx=0x7faa8c009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 -- 172.21.15.86:0/960932022 shutdown_connections 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 --2- 172.21.15.86:0/960932022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa90105ae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 -- 172.21.15.86:0/960932022 >> 172.21.15.86:0/960932022 conn(0x7faa900fb210 msgr2=0x7faa900fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 -- 172.21.15.86:0/960932022 shutdown_connections 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 -- 172.21.15.86:0/960932022 wait complete. 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.820+0000 7faa981f9700 1 Processor -- start 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa981f9700 1 -- start start 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa981f9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa901012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa981f9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa90103250 con 0x7faa90105700 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa95f95700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa901012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa95f95700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa901012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:38532/0 (socket says 172.21.15.86:38532) 2024-02-19T18:57:11.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa95f95700 1 -- 172.21.15.86:0/3917050531 learned_addr learned my addr 172.21.15.86:0/3917050531 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa95f95700 1 -- 172.21.15.86:0/3917050531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0095f0 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.821+0000 7faa95f95700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa901012d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faa8c005b40 tx=0x7faa8c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.822+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c01d070 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.822+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faa8c0044d0 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.822+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c003930 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.822+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa90101810 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.822+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa90101c90 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.822+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7faa8c003ba0 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.823+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa900623c0 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.823+0000 7faa86ffd700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa7c038500 0x7faa7c03a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.823+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7faa8c04c410 con 0x7faa90105700 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.823+0000 7faa95794700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa7c038500 0x7faa7c03a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.823+0000 7faa95794700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa7c038500 0x7faa7c03a9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faa80006fd0 tx=0x7faa80006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:11.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:11.827+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faa8c02a540 con 0x7faa90105700 2024-02-19T18:57:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.107+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa90102120 con 0x7faa90105700 2024-02-19T18:57:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.108+0000 7faa86ffd700 1 -- 172.21.15.86:0/3917050531 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7faa8c02a360 con 0x7faa90105700 2024-02-19T18:57:12.125 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:12.126 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa7c038500 msgr2=0x7faa7c03a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa7c038500 0x7faa7c03a9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faa80006fd0 tx=0x7faa80006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 msgr2=0x7faa901012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa901012d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faa8c005b40 tx=0x7faa8c004c80 comp rx=0 tx=0).stop 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 shutdown_connections 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa7c038500 0x7faa7c03a9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 --2- 172.21.15.86:0/3917050531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa90105700 0x7faa901012d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 >> 172.21.15.86:0/3917050531 conn(0x7faa900fb210 msgr2=0x7faa90107b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 shutdown_connections 2024-02-19T18:57:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:12.110+0000 7faa981f9700 1 -- 172.21.15.86:0/3917050531 wait complete. 2024-02-19T18:57:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:12.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:12.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:12.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:57:12.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T18:57:12.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:12.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:12 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3917050531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:13.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:13 smithi039 ceph-mon[28988]: Deploying daemon ceph-exporter.smithi086 on smithi086 2024-02-19T18:57:13.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:13.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:14.111 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:15.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 -- 172.21.15.86:0/3699076061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 msgr2=0x7f01700ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:15.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 --2- 172.21.15.86:0/3699076061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01700ff500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0160009910 tx=0x7f0160009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:15.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 -- 172.21.15.86:0/3699076061 shutdown_connections 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 --2- 172.21.15.86:0/3699076061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01700ff500 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 -- 172.21.15.86:0/3699076061 >> 172.21.15.86:0/3699076061 conn(0x7f0170075e80 msgr2=0x7f0170076290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 -- 172.21.15.86:0/3699076061 shutdown_connections 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.631+0000 7f0175207700 1 -- 172.21.15.86:0/3699076061 wait complete. 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f0175207700 1 Processor -- start 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f0175207700 1 -- start start 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f0175207700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01701000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f0175207700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0170102020 con 0x7f01700ff120 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f016ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01701000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f016ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01701000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:38556/0 (socket says 172.21.15.86:38556) 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.632+0000 7f016ed9d700 1 -- 172.21.15.86:0/3498301156 learned_addr learned my addr 172.21.15.86:0/3498301156 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f016ed9d700 1 -- 172.21.15.86:0/3498301156 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600095f0 con 0x7f01700ff120 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f016ed9d700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01701000a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f01600092e0 tx=0x7f0160004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016001d070 con 0x7f01700ff120 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01701005e0 con 0x7f01700ff120 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f016000be10 con 0x7f01700ff120 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0160022e10 con 0x7f01700ff120 2024-02-19T18:57:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.633+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0170100a60 con 0x7f01700ff120 2024-02-19T18:57:15.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.634+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f016000f460 con 0x7f01700ff120 2024-02-19T18:57:15.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.634+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0170108dd0 con 0x7f01700ff120 2024-02-19T18:57:15.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.634+0000 7f0157fff700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0158038460 0x7f015803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:15.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.634+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f016004c4e0 con 0x7f01700ff120 2024-02-19T18:57:15.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.634+0000 7f016e59c700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0158038460 0x7f015803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:15.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.635+0000 7f016e59c700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0158038460 0x7f015803a920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0164006fd0 tx=0x7f0164006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:15.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.639+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f016002a7d0 con 0x7f01700ff120 2024-02-19T18:57:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.910+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01700623c0 con 0x7f01700ff120 2024-02-19T18:57:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.911+0000 7f0157fff700 1 -- 172.21.15.86:0/3498301156 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0160027030 con 0x7f01700ff120 2024-02-19T18:57:15.930 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:15.931 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0158038460 msgr2=0x7f015803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0158038460 0x7f015803a920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0164006fd0 tx=0x7f0164006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 msgr2=0x7f01701000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01701000a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f01600092e0 tx=0x7f0160004c80 comp rx=0 tx=0).stop 2024-02-19T18:57:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 shutdown_connections 2024-02-19T18:57:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0158038460 0x7f015803a920 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 --2- 172.21.15.86:0/3498301156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01700ff120 0x7f01701000a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 >> 172.21.15.86:0/3498301156 conn(0x7f0170075e80 msgr2=0x7f0170104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 shutdown_connections 2024-02-19T18:57:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:15.913+0000 7f0175207700 1 -- 172.21.15.86:0/3498301156 wait complete. 2024-02-19T18:57:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:15.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:17.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3498301156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:17.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:17.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:17.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:17.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:57:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:57:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:17.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:17.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:17.870 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:18.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:17 smithi039 ceph-mon[28988]: Deploying daemon crash.smithi086 on smithi086 2024-02-19T18:57:19.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 -- 172.21.15.86:0/2652462045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 msgr2=0x7f2028101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 --2- 172.21.15.86:0/2652462045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028101dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2018009910 tx=0x7f2018009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:19.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 -- 172.21.15.86:0/2652462045 shutdown_connections 2024-02-19T18:57:19.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 --2- 172.21.15.86:0/2652462045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028101dc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 -- 172.21.15.86:0/2652462045 >> 172.21.15.86:0/2652462045 conn(0x7f20280fb250 msgr2=0x7f20280fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 -- 172.21.15.86:0/2652462045 shutdown_connections 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.436+0000 7f202ccdf700 1 -- 172.21.15.86:0/2652462045 wait complete. 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202ccdf700 1 Processor -- start 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202ccdf700 1 -- start start 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202ccdf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202ccdf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028196310 con 0x7f20281019e0 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:38580/0 (socket says 172.21.15.86:38580) 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.437+0000 7f202659c700 1 -- 172.21.15.86:0/1789035283 learned_addr learned my addr 172.21.15.86:0/1789035283 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f202659c700 1 -- 172.21.15.86:0/1789035283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20180095f0 con 0x7f20281019e0 2024-02-19T18:57:19.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f202659c700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028195dd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2018005b40 tx=0x7f2018004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f201801d070 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2028196510 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20281968d0 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f201800be10 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.438+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2018022b60 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.439+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2008005320 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.439+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f2018022cc0 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.439+0000 7f20177fe700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f20100384b0 0x7f201003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.439+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f201804c3e0 con 0x7f20281019e0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.440+0000 7f2025d9b700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f20100384b0 0x7f201003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:19.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.440+0000 7f2025d9b700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f20100384b0 0x7f201003a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f201c006fd0 tx=0x7f201c006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:19.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.445+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f201802ab50 con 0x7f20281019e0 2024-02-19T18:57:19.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.738+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2008005190 con 0x7f20281019e0 2024-02-19T18:57:19.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.739+0000 7f20177fe700 1 -- 172.21.15.86:0/1789035283 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f201802ab50 con 0x7f20281019e0 2024-02-19T18:57:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:19.757 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f20100384b0 msgr2=0x7f201003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f20100384b0 0x7f201003a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f201c006fd0 tx=0x7f201c006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 msgr2=0x7f2028195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028195dd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2018005b40 tx=0x7f2018004c80 comp rx=0 tx=0).stop 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 shutdown_connections 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f20100384b0 0x7f201003a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 --2- 172.21.15.86:0/1789035283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20281019e0 0x7f2028195dd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 >> 172.21.15.86:0/1789035283 conn(0x7f20280fb250 msgr2=0x7f2028105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 shutdown_connections 2024-02-19T18:57:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:19.741+0000 7f202ccdf700 1 -- 172.21.15.86:0/1789035283 wait complete. 2024-02-19T18:57:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:20.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:19 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/1789035283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:21.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:21.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:21.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:21.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:21.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:21.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:21.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: Deploying daemon node-exporter.smithi086 on smithi086 2024-02-19T18:57:21.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:21.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:23.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:23 smithi039 ceph-mon[28988]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 -- 172.21.15.86:0/662447192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 msgr2=0x7f6fa8105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 --2- 172.21.15.86:0/662447192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8105ae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009910 tx=0x7f6f98009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 -- 172.21.15.86:0/662447192 shutdown_connections 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 --2- 172.21.15.86:0/662447192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8105ae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 -- 172.21.15.86:0/662447192 >> 172.21.15.86:0/662447192 conn(0x7f6fa80fb210 msgr2=0x7f6fa80fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 -- 172.21.15.86:0/662447192 shutdown_connections 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.139+0000 7f6fad9d7700 1 -- 172.21.15.86:0/662447192 wait complete. 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.140+0000 7f6fad9d7700 1 Processor -- start 2024-02-19T18:57:24.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.140+0000 7f6fad9d7700 1 -- start start 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.140+0000 7f6fad9d7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.140+0000 7f6fad9d7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa8196260 con 0x7f6fa8105700 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fa6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fa6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:45634/0 (socket says 172.21.15.86:45634) 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fa6ffd700 1 -- 172.21.15.86:0/365777516 learned_addr learned my addr 172.21.15.86:0/365777516 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fa6ffd700 1 -- 172.21.15.86:0/365777516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980095f0 con 0x7f6fa8105700 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fa6ffd700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8195d20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f98005b40 tx=0x7f6f98004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f9801d070 con 0x7f6fa8105700 2024-02-19T18:57:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa8196460 con 0x7f6fa8105700 2024-02-19T18:57:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa81968e0 con 0x7f6fa8105700 2024-02-19T18:57:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6f9800be10 con 0x7f6fa8105700 2024-02-19T18:57:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.141+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f98022b60 con 0x7f6fa8105700 2024-02-19T18:57:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.142+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f94005320 con 0x7f6fa8105700 2024-02-19T18:57:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.142+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f6f98022cc0 con 0x7f6fa8105700 2024-02-19T18:57:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.143+0000 7f6fac9d5700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6f900384b0 0x7f6f9003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.143+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6f9804c2e0 con 0x7f6fa8105700 2024-02-19T18:57:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.143+0000 7f6fa67fc700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6f900384b0 0x7f6f9003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.143+0000 7f6fa67fc700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6f900384b0 0x7f6f9003a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c006fd0 tx=0x7f6f9c006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.148+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6f980217c0 con 0x7f6fa8105700 2024-02-19T18:57:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.419+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f94005190 con 0x7f6fa8105700 2024-02-19T18:57:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.419+0000 7f6fac9d5700 1 -- 172.21.15.86:0/365777516 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6f980213d0 con 0x7f6fa8105700 2024-02-19T18:57:24.437 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:24.437 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6f900384b0 msgr2=0x7f6f9003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6f900384b0 0x7f6f9003a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c006fd0 tx=0x7f6f9c006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 msgr2=0x7f6fa8195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8195d20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6f98005b40 tx=0x7f6f98004c80 comp rx=0 tx=0).stop 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 shutdown_connections 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6f900384b0 0x7f6f9003a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 --2- 172.21.15.86:0/365777516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fa8105700 0x7f6fa8195d20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 >> 172.21.15.86:0/365777516 conn(0x7f6fa80fb210 msgr2=0x7f6fa8107b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 shutdown_connections 2024-02-19T18:57:24.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:24.421+0000 7f6fad9d7700 1 -- 172.21.15.86:0/365777516 wait complete. 2024-02-19T18:57:24.440 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:25.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:24 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/365777516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:25.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:24 smithi039 ceph-mon[28988]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:26.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:26.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:26.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:26.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:26.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:26.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:26.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:57:26.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T18:57:26.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:57:26.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:26.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: Deploying daemon mgr.smithi086.huubbu on smithi086 2024-02-19T18:57:26.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:27.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:27 smithi039 ceph-mon[28988]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:28.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 -- 172.21.15.86:0/3045949589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 msgr2=0x7fc090105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:28.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 --2- 172.21.15.86:0/3045949589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090105b60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009910 tx=0x7fc08c009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:28.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 -- 172.21.15.86:0/3045949589 shutdown_connections 2024-02-19T18:57:28.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 --2- 172.21.15.86:0/3045949589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090105b60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 -- 172.21.15.86:0/3045949589 >> 172.21.15.86:0/3045949589 conn(0x7fc090075290 msgr2=0x7fc0900756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 -- 172.21.15.86:0/3045949589 shutdown_connections 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.379+0000 7fc0972b6700 1 -- 172.21.15.86:0/3045949589 wait complete. 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc0972b6700 1 Processor -- start 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc0972b6700 1 -- start start 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc0972b6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc0972b6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901962a0 con 0x7fc090105780 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc095052700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc095052700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.86:45666/0 (socket says 172.21.15.86:45666) 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc095052700 1 -- 172.21.15.86:0/2785150535 learned_addr learned my addr 172.21.15.86:0/2785150535 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.380+0000 7fc095052700 1 -- 172.21.15.86:0/2785150535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c0095f0 con 0x7fc090105780 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.381+0000 7fc095052700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090195d60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc08c005b40 tx=0x7fc08c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.381+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c01d070 con 0x7fc090105780 2024-02-19T18:57:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.381+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0901964a0 con 0x7fc090105780 2024-02-19T18:57:28.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.381+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc08c0044d0 con 0x7fc090105780 2024-02-19T18:57:28.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.381+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c003930 con 0x7fc090105780 2024-02-19T18:57:28.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.381+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc090196860 con 0x7fc090105780 2024-02-19T18:57:28.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.382+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0900623c0 con 0x7fc090105780 2024-02-19T18:57:28.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.382+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fc08c00b960 con 0x7fc090105780 2024-02-19T18:57:28.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.382+0000 7fc0867fc700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc07c0384b0 0x7fc07c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:28.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.382+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc08c04c3d0 con 0x7fc090105780 2024-02-19T18:57:28.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.382+0000 7fc094851700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc07c0384b0 0x7fc07c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:28.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.383+0000 7fc094851700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc07c0384b0 0x7fc07c03a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc080006fd0 tx=0x7fc080006e40 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:28.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.388+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc08c02a540 con 0x7fc090105780 2024-02-19T18:57:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.656+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc09019a850 con 0x7fc090105780 2024-02-19T18:57:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.657+0000 7fc0867fc700 1 -- 172.21.15.86:0/2785150535 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc08c02a360 con 0x7fc090105780 2024-02-19T18:57:28.675 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:28.675 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:54:40.954257Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:57:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.658+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc07c0384b0 msgr2=0x7fc07c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.658+0000 7fc0972b6700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc07c0384b0 0x7fc07c03a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc080006fd0 tx=0x7fc080006e40 comp rx=0 tx=0).stop 2024-02-19T18:57:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.658+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 msgr2=0x7fc090195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.658+0000 7fc0972b6700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090195d60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc08c005b40 tx=0x7fc08c004c80 comp rx=0 tx=0).stop 2024-02-19T18:57:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.659+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 shutdown_connections 2024-02-19T18:57:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.659+0000 7fc0972b6700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc07c0384b0 0x7fc07c03a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.659+0000 7fc0972b6700 1 --2- 172.21.15.86:0/2785150535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc090105780 0x7fc090195d60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.659+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 >> 172.21.15.86:0/2785150535 conn(0x7fc090075290 msgr2=0x7fc090067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.659+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 shutdown_connections 2024-02-19T18:57:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:28.659+0000 7fc0972b6700 1 -- 172.21.15.86:0/2785150535 wait complete. 2024-02-19T18:57:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-02-19T18:57:28.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:28 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/2785150535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:29.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:29.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:29.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:29.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:29.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:29.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:57:29.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:30.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:57:30.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:57:30.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:30 smithi039 ceph-mon[28988]: Deploying daemon mon.smithi086 on smithi086 2024-02-19T18:57:30.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:31.219 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:31.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:31 smithi039 ceph-mon[28988]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:31.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:38.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: mon.smithi039 calling monitor election 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: mon.smithi086 calling monitor election 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/484302342' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:38.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: mon.smithi039 is new leader, mons smithi039,smithi086 in quorum (ranks 0,1) 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: monmap e2: 2 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: fsmap 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: mgrmap e17: smithi039.ydmpuh(active, since 37s) 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: Standby manager daemon smithi086.huubbu started 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/484302342' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: overall HEALTH_OK 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/484302342' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/484302342' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:38.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:38.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:38 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:39.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:39 smithi039 ceph-mon[28988]: mgrmap e18: smithi039.ydmpuh(active, since 37s), standbys: smithi086.huubbu 2024-02-19T18:57:39.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T18:57:39.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:39.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:39 smithi039 ceph-mon[28988]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:39.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:39.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:39 smithi086 ceph-mon[37629]: mgrmap e18: smithi039.ydmpuh(active, since 37s), standbys: smithi086.huubbu 2024-02-19T18:57:39.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T18:57:39.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:39.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:39 smithi086 ceph-mon[37629]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:39.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 -- 172.21.15.86:0/786813113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 msgr2=0x7f2d280086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 --2- 172.21.15.86:0/786813113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d280086b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2d400ffd90 tx=0x7f2d3000fe90 comp rx=0 tx=0).stop 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 -- 172.21.15.86:0/786813113 shutdown_connections 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 --2- 172.21.15.86:0/786813113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d280086b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 -- 172.21.15.86:0/786813113 >> 172.21.15.86:0/786813113 conn(0x7f2d40075df0 msgr2=0x7f2d40076200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 -- 172.21.15.86:0/786813113 shutdown_connections 2024-02-19T18:57:39.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.830+0000 7f2d44a49700 1 -- 172.21.15.86:0/786813113 wait complete. 2024-02-19T18:57:39.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.831+0000 7f2d44a49700 1 Processor -- start 2024-02-19T18:57:39.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.831+0000 7f2d44a49700 1 -- start start 2024-02-19T18:57:39.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.831+0000 7f2d44a49700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d40195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.831+0000 7f2d44a49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.831+0000 7f2d44a49700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d40196a00 con 0x7f2d28006240 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.831+0000 7f2d44a49700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d40196b70 con 0x7f2d401963e0 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38216/0 (socket says 172.21.15.86:38216) 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 -- 172.21.15.86:0/244679153 learned_addr learned my addr 172.21.15.86:0/244679153 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 -- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 msgr2=0x7f2d4019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 -- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 msgr2=0x7f2d4019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3dd9b700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3e59c700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d40195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3e59c700 1 -- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 msgr2=0x7f2d4019a850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3e59c700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3e59c700 1 -- 172.21.15.86:0/244679153 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3000f8e0 con 0x7f2d28006240 2024-02-19T18:57:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.832+0000 7f2d3e59c700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d40195ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2d400ffd90 tx=0x7f2d3001d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.833+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d30005da0 con 0x7f2d28006240 2024-02-19T18:57:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.833+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2d3001dc10 con 0x7f2d28006240 2024-02-19T18:57:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.833+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d30011830 con 0x7f2d28006240 2024-02-19T18:57:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.833+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4019ae50 con 0x7f2d28006240 2024-02-19T18:57:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.833+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4019b3a0 con 0x7f2d28006240 2024-02-19T18:57:39.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.833+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d400623c0 con 0x7f2d28006240 2024-02-19T18:57:39.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.834+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f2d30005430 con 0x7f2d28006240 2024-02-19T18:57:39.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.835+0000 7f2d2f7fe700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d2006c5c0 0x7f2d2006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:39.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.835+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f2d3008db20 con 0x7f2d28006240 2024-02-19T18:57:39.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.835+0000 7f2d3dd9b700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d2006c5c0 0x7f2d2006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:39.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.835+0000 7f2d3dd9b700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d2006c5c0 0x7f2d2006ea80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009180 tx=0x7f2d34008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:39.839+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2d30059590 con 0x7f2d28006240 2024-02-19T18:57:40.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.112+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d40197200 con 0x7f2d28006240 2024-02-19T18:57:40.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.112+0000 7f2d2f7fe700 1 -- 172.21.15.86:0/244679153 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1018 (secure 0 0 0) 0x7f2d3005cbb0 con 0x7f2d28006240 2024-02-19T18:57:40.130 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-02-19T18:57:40.130 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","modified":"2024-02-19T18:57:33.317972Z","created":"2024-02-19T18:54:40.954257Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi039","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi086","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-19T18:57:40.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d2006c5c0 msgr2=0x7f2d2006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:40.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d2006c5c0 0x7f2d2006ea80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009180 tx=0x7f2d34008040 comp rx=0 tx=0).stop 2024-02-19T18:57:40.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 msgr2=0x7f2d40195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:40.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d40195ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2d400ffd90 tx=0x7f2d3001d9b0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 shutdown_connections 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d2006c5c0 0x7f2d2006ea80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d28006240 0x7f2d40195ea0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 --2- 172.21.15.86:0/244679153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d401963e0 0x7f2d4019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.114+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 >> 172.21.15.86:0/244679153 conn(0x7f2d40075df0 msgr2=0x7f2d400fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.115+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 shutdown_connections 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:57:40.115+0000 7f2d44a49700 1 -- 172.21.15.86:0/244679153 wait complete. 2024-02-19T18:57:40.133 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2024-02-19T18:57:40.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:40 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/244679153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:40.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:40 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/244679153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:57:40.965 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-19T18:57:40.965 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph config generate-minimal-conf 2024-02-19T18:57:41.195 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:57:41.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:57:41.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:57:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:42.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:42 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T18:57:42.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:42 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T18:57:42.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:42 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:42.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:42 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:42.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:42 smithi039 ceph-mon[28988]: Reconfiguring mon.smithi039 (unknown last config time)... 2024-02-19T18:57:42.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:42 smithi039 ceph-mon[28988]: Reconfiguring daemon mon.smithi039 on smithi039 2024-02-19T18:57:43.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:42 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T18:57:43.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:42 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T18:57:43.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:42 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:43.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:42 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:57:43.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:42 smithi086 ceph-mon[37629]: Reconfiguring mon.smithi039 (unknown last config time)... 2024-02-19T18:57:43.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:42 smithi086 ceph-mon[37629]: Reconfiguring daemon mon.smithi039 on smithi039 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 -- 172.21.15.39:0/3401295601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60106820 msgr2=0x7fea60106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 --2- 172.21.15.39:0/3401295601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60106820 0x7fea60106c00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea5000b0a0 tx=0x7fea5000b3b0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 -- 172.21.15.39:0/3401295601 shutdown_connections 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 --2- 172.21.15.39:0/3401295601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60106820 0x7fea60106c00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 -- 172.21.15.39:0/3401295601 >> 172.21.15.39:0/3401295601 conn(0x7fea600fc350 msgr2=0x7fea600fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 -- 172.21.15.39:0/3401295601 shutdown_connections 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 -- 172.21.15.39:0/3401295601 wait complete. 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.444+0000 7fea678e1700 1 Processor -- start 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 -- start start 2024-02-19T18:57:43.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea60106820 0x7fea60195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 0x7fea6019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea601969a0 con 0x7fea60196380 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea60196b10 con 0x7fea60106820 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 0x7fea6019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 0x7fea6019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32882/0 (socket says 172.21.15.39:32882) 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 -- 172.21.15.39:0/3661021313 learned_addr learned my addr 172.21.15.39:0/3661021313 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 -- 172.21.15.39:0/3661021313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea60106820 msgr2=0x7fea60195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea60106820 0x7fea60195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 -- 172.21.15.39:0/3661021313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea50009d00 con 0x7fea60196380 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea64e7c700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 0x7fea6019a7f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fea5400b700 tx=0x7fea5400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea54011800 con 0x7fea60196380 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fea54011e10 con 0x7fea60196380 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6019adf0 con 0x7fea60196380 2024-02-19T18:57:43.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea5400f2e0 con 0x7fea60196380 2024-02-19T18:57:43.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.445+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6019b340 con 0x7fea60196380 2024-02-19T18:57:43.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.446+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6019b670 con 0x7fea60196380 2024-02-19T18:57:43.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.447+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7fea54010380 con 0x7fea60196380 2024-02-19T18:57:43.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.447+0000 7fea5e7fc700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fea4806c5c0 0x7fea4806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:43.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.447+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fea5408a330 con 0x7fea60196380 2024-02-19T18:57:43.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.447+0000 7fea6567d700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fea4806c5c0 0x7fea4806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:43.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.448+0000 7fea6567d700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fea4806c5c0 0x7fea4806ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea50009480 tx=0x7fea5000b7b0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:43.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.452+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fea54055bf0 con 0x7fea60196380 2024-02-19T18:57:43.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.655+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fea601a0a90 con 0x7fea60196380 2024-02-19T18:57:43.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.655+0000 7fea5e7fc700 1 -- 172.21.15.39:0/3661021313 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7fea540150f0 con 0x7fea60196380 2024-02-19T18:57:43.674 INFO:teuthology.orchestra.run.smithi039.stdout:# minimal ceph.conf for 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:57:43.674 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-02-19T18:57:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T18:57:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: mon_host = [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-02-19T18:57:43.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fea4806c5c0 msgr2=0x7fea4806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:43.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fea4806c5c0 0x7fea4806ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fea50009480 tx=0x7fea5000b7b0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 msgr2=0x7fea6019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 0x7fea6019a7f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fea5400b700 tx=0x7fea5400bac0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 shutdown_connections 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fea4806c5c0 0x7fea4806ea80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea60106820 0x7fea60195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 --2- 172.21.15.39:0/3661021313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea60196380 0x7fea6019a7f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 >> 172.21.15.39:0/3661021313 conn(0x7fea600fc350 msgr2=0x7fea60102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 shutdown_connections 2024-02-19T18:57:43.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:43.657+0000 7fea678e1700 1 -- 172.21.15.39:0/3661021313 wait complete. 2024-02-19T18:57:44.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:43 smithi086 ceph-mon[37629]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:44.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:43 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3661021313' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:44.035 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:43 smithi039 ceph-mon[28988]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:44.035 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:43 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3661021313' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:44.519 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-19T18:57:44.520 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:57:44.520 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T18:57:44.550 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:57:44.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:57:44.616 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:57:44.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T18:57:44.651 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:57:44.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:57:44.722 INFO:tasks.cephadm:Deploying OSDs... 2024-02-19T18:57:44.722 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T18:57:44.722 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T18:57:44.737 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T18:57:44.737 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-02-19 18:56:52.161750084 +0000 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-02-19 18:48:39.309369212 +0000 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-02-19 18:48:39.309369212 +0000 2024-02-19T18:57:44.792 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-02-19 18:48:39.309369212 +0000 2024-02-19T18:57:44.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T18:57:44.858 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-02-19T18:57:44.859 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-02-19T18:57:44.859 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000263102 s, 1.9 MB/s 2024-02-19T18:57:44.860 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T18:57:44.915 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-02-19T18:57:44.972 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T18:57:44.972 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:44.972 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-19T18:57:44.972 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:44.973 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:44.973 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-02-19 18:56:52.162750064 +0000 2024-02-19T18:57:44.973 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-02-19 18:48:39.667362268 +0000 2024-02-19T18:57:44.973 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-02-19 18:48:39.667362268 +0000 2024-02-19T18:57:44.973 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-02-19 18:48:39.667362268 +0000 2024-02-19T18:57:44.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T18:57:45.039 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-02-19T18:57:45.039 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-02-19T18:57:45.039 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000317973 s, 1.6 MB/s 2024-02-19T18:57:45.040 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T18:57:45.097 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-02-19T18:57:45.152 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T18:57:45.152 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:45.152 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-19T18:57:45.152 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-02-19 18:56:52.162750064 +0000 2024-02-19T18:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-02-19 18:48:40.017355479 +0000 2024-02-19T18:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-02-19 18:48:40.017355479 +0000 2024-02-19T18:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-02-19 18:48:40.017355479 +0000 2024-02-19T18:57:45.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: Reconfiguring mgr.smithi039.ydmpuh (unknown last config time)... 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: Reconfiguring daemon mgr.smithi039.ydmpuh on smithi039 2024-02-19T18:57:45.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:44 smithi039 ceph-mon[28988]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:45.218 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-02-19T18:57:45.218 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-02-19T18:57:45.218 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000322675 s, 1.6 MB/s 2024-02-19T18:57:45.219 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T18:57:45.274 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: Reconfiguring mgr.smithi039.ydmpuh (unknown last config time)... 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: Reconfiguring daemon mgr.smithi039.ydmpuh on smithi039 2024-02-19T18:57:45.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:44 smithi086 ceph-mon[37629]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:45.328 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T18:57:45.328 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:45.328 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-19T18:57:45.329 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:45.329 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:45.329 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-02-19 18:56:52.162750064 +0000 2024-02-19T18:57:45.329 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-02-19 18:48:40.387348302 +0000 2024-02-19T18:57:45.329 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-02-19 18:48:40.387348302 +0000 2024-02-19T18:57:45.329 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-02-19 18:48:40.387348302 +0000 2024-02-19T18:57:45.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T18:57:45.393 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-02-19T18:57:45.394 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-02-19T18:57:45.394 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000323353 s, 1.6 MB/s 2024-02-19T18:57:45.395 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T18:57:45.450 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T18:57:45.450 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T18:57:45.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T18:57:45.469 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-02-19T18:57:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T18:57:45.525 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:45.525 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-19T18:57:45.526 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:45.526 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:45.526 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-02-19 18:57:32.316873530 +0000 2024-02-19T18:57:45.526 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-02-19 18:48:36.582605251 +0000 2024-02-19T18:57:45.526 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-02-19 18:48:36.582605251 +0000 2024-02-19T18:57:45.526 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-02-19 18:48:36.582605251 +0000 2024-02-19T18:57:45.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T18:57:45.594 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-02-19T18:57:45.594 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-02-19T18:57:45.594 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000245234 s, 2.1 MB/s 2024-02-19T18:57:45.595 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T18:57:45.653 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-02-19T18:57:45.708 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-02-19 18:57:32.317873509 +0000 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-02-19 18:48:36.934597933 +0000 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-02-19 18:48:36.934597933 +0000 2024-02-19T18:57:45.709 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-02-19 18:48:36.934597933 +0000 2024-02-19T18:57:45.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T18:57:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-02-19T18:57:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-02-19T18:57:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.00020633 s, 2.5 MB/s 2024-02-19T18:57:45.777 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T18:57:45.834 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-02-19T18:57:45.889 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T18:57:45.889 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:45.889 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-19T18:57:45.889 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:45.889 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:45.889 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-02-19 18:57:32.317873509 +0000 2024-02-19T18:57:45.890 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-02-19 18:48:37.297590386 +0000 2024-02-19T18:57:45.890 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-02-19 18:48:37.297590386 +0000 2024-02-19T18:57:45.890 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-02-19 18:48:37.297590386 +0000 2024-02-19T18:57:45.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T18:57:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-02-19T18:57:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-02-19T18:57:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000225198 s, 2.3 MB/s 2024-02-19T18:57:45.957 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T18:57:46.013 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-02-19T18:57:46.068 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-02-19 18:57:32.317873509 +0000 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-02-19 18:48:37.645583152 +0000 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-02-19 18:48:37.645583152 +0000 2024-02-19T18:57:46.069 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-02-19 18:48:37.645583152 +0000 2024-02-19T18:57:46.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T18:57:46.133 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-02-19T18:57:46.134 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-02-19T18:57:46.134 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000280475 s, 1.8 MB/s 2024-02-19T18:57:46.135 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T18:57:46.190 INFO:tasks.cephadm:Deploying osd.0 on smithi039 with /dev/vg_nvme/lv_4... 2024-02-19T18:57:46.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T18:57:46.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:46.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:46.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:46.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: Reconfiguring ceph-exporter.smithi039 (monmap changed)... 2024-02-19T18:57:46.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:57:46.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:46.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:45 smithi039 ceph-mon[28988]: Reconfiguring daemon ceph-exporter.smithi039 on smithi039 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: Reconfiguring ceph-exporter.smithi039 (monmap changed)... 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:45 smithi086 ceph-mon[37629]: Reconfiguring daemon ceph-exporter.smithi039 on smithi039 2024-02-19T18:57:47.144 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:46 smithi039 ceph-mon[28988]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:47.162 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:57:47.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:46 smithi086 ceph-mon[37629]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:48.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:48 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:48.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:48 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:48.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:48 smithi039 ceph-mon[28988]: Reconfiguring crash.smithi039 (monmap changed)... 2024-02-19T18:57:48.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:48 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:57:48.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:48 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:48.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:48 smithi039 ceph-mon[28988]: Reconfiguring daemon crash.smithi039 on smithi039 2024-02-19T18:57:48.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:48 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:48.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:48 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:48.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:48 smithi086 ceph-mon[37629]: Reconfiguring crash.smithi039 (monmap changed)... 2024-02-19T18:57:48.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:48 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:57:48.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:48 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:48.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:48 smithi086 ceph-mon[37629]: Reconfiguring daemon crash.smithi039 on smithi039 2024-02-19T18:57:48.994 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T18:57:48.994 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T18:57:48.994 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-02-19T18:57:48.995 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-02-19T18:57:48.995 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130403 s, 804 MB/s 2024-02-19T18:57:48.995 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-02-19T18:57:49.494 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:49 smithi039 ceph-mon[28988]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:49.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:49 smithi086 ceph-mon[37629]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:50.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_4 2024-02-19T18:57:51.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:51.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:51.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:50 smithi039 ceph-mon[28988]: Reconfiguring alertmanager.smithi039 (dependencies changed)... 2024-02-19T18:57:51.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:50 smithi039 ceph-mon[28988]: Reconfiguring daemon alertmanager.smithi039 on smithi039 2024-02-19T18:57:51.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:51.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:51.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:50 smithi086 ceph-mon[37629]: Reconfiguring alertmanager.smithi039 (dependencies changed)... 2024-02-19T18:57:51.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:50 smithi086 ceph-mon[37629]: Reconfiguring daemon alertmanager.smithi039 on smithi039 2024-02-19T18:57:52.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:51 smithi039 ceph-mon[28988]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:52.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:51 smithi086 ceph-mon[37629]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:52.857 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:57:53.916 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:53 smithi039 ceph-mon[28988]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:54.084 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:53 smithi086 ceph-mon[37629]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2674871270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58104070 msgr2=0x7f5e58106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2674871270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58104070 0x7f5e58106460 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009910 tx=0x7f5e4c009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2674871270 shutdown_connections 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2674871270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58104070 0x7f5e58106460 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2674871270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e58101740 0x7f5e58103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2674871270 >> 172.21.15.39:0/2674871270 conn(0x7f5e580fb150 msgr2=0x7f5e580fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2674871270 shutdown_connections 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2674871270 wait complete. 2024-02-19T18:57:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 Processor -- start 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.954+0000 7f5e5e86e700 1 -- start start 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e5e86e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 0x7f5e58196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e5e86e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e58104070 0x7f5e581969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e5e86e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58196fc0 con 0x7f5e58101740 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e5e86e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58197100 con 0x7f5e58104070 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 0x7f5e58196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 0x7f5e58196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47480/0 (socket says 172.21.15.39:47480) 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 -- 172.21.15.39:0/2668322514 learned_addr learned my addr 172.21.15.39:0/2668322514 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 -- 172.21.15.39:0/2668322514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e58104070 msgr2=0x7f5e581969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e577fe700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e58104070 0x7f5e581969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e58104070 0x7f5e581969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 -- 172.21.15.39:0/2668322514 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c0095f0 con 0x7f5e58101740 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e57fff700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 0x7f5e58196460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800c8a0 tx=0x7f5e4800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:54.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4800cea0 con 0x7f5e58101740 2024-02-19T18:57:54.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5e48004830 con 0x7f5e58101740 2024-02-19T18:57:54.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5819bbb0 con 0x7f5e58101740 2024-02-19T18:57:54.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.955+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e480055b0 con 0x7f5e58101740 2024-02-19T18:57:54.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.956+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5819c100 con 0x7f5e58101740 2024-02-19T18:57:54.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.957+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e58190620 con 0x7f5e58101740 2024-02-19T18:57:54.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.957+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f5e48007ad0 con 0x7f5e58101740 2024-02-19T18:57:54.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.957+0000 7f5e557fa700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e4006c460 0x7f5e4006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:54.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.957+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f5e4808a430 con 0x7f5e58101740 2024-02-19T18:57:54.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.958+0000 7f5e577fe700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e4006c460 0x7f5e4006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:54.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.958+0000 7f5e577fe700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e4006c460 0x7f5e4006e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00b5c0 tx=0x7f5e4c005960 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:54.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:54.963+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5e48059780 con 0x7f5e58101740 2024-02-19T18:57:55.089 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:54 smithi039 ceph-mon[28988]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:55.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:57:55.175+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e580611d0 con 0x7f5e4006c460 2024-02-19T18:57:55.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:54 smithi086 ceph-mon[37629]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:56.054 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:55 smithi039 ceph-mon[28988]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:56.054 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:56.054 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:56.054 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:56.279 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:55 smithi086 ceph-mon[37629]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:56.279 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:56.279 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:56.279 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:57.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:57 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:57.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:57 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:57.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:57 smithi039 ceph-mon[28988]: Reconfiguring grafana.smithi039 (dependencies changed)... 2024-02-19T18:57:57.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:57 smithi039 ceph-mon[28988]: Reconfiguring daemon grafana.smithi039 on smithi039 2024-02-19T18:57:57.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:57 smithi039 ceph-mon[28988]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:57 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:57 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:57:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:57 smithi086 ceph-mon[37629]: Reconfiguring grafana.smithi039 (dependencies changed)... 2024-02-19T18:57:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:57 smithi086 ceph-mon[37629]: Reconfiguring daemon grafana.smithi039 on smithi039 2024-02-19T18:57:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:57 smithi086 ceph-mon[37629]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:57:59.864 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:57:59 smithi039 ceph-mon[28988]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:00.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:57:59 smithi086 ceph-mon[37629]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:00.909 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:01.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:01.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:01 smithi039 ceph-mon[28988]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:01.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:01.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:01 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:02.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:01 smithi086 ceph-mon[37629]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:02.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:01 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:02.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:01 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:03.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: Reconfiguring prometheus.smithi039 (dependencies changed)... 2024-02-19T18:58:03.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: Reconfiguring daemon prometheus.smithi039 on smithi039 2024-02-19T18:58:03.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3443733726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f217d9-d1a8-4658-b693-873f74d42408"}]: dispatch 2024-02-19T18:58:03.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3443733726' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f217d9-d1a8-4658-b693-873f74d42408"}]': finished 2024-02-19T18:58:03.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: osdmap e6: 1 total, 0 up, 1 in 2024-02-19T18:58:03.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:03.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:02 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3823912639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: Reconfiguring prometheus.smithi039 (dependencies changed)... 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: Reconfiguring daemon prometheus.smithi039 on smithi039 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3443733726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91f217d9-d1a8-4658-b693-873f74d42408"}]: dispatch 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3443733726' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91f217d9-d1a8-4658-b693-873f74d42408"}]': finished 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: osdmap e6: 1 total, 0 up, 1 in 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:02 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3823912639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:04.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:03 smithi039 ceph-mon[28988]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:04.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:03 smithi086 ceph-mon[37629]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:05.146 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:05 smithi039 ceph-mon[28988]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:05.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:05 smithi086 ceph-mon[37629]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:07.846 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:07 smithi039 ceph-mon[28988]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:08.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:07 smithi086 ceph-mon[37629]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:09.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:09 smithi039 ceph-mon[28988]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:10.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:09 smithi086 ceph-mon[37629]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:11.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:11 smithi039 ceph-mon[28988]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:12.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:11 smithi086 ceph-mon[37629]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:13.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:13 smithi086 ceph-mon[37629]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:13.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:13.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:13.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:58:13.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:13.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:13 smithi039 ceph-mon[28988]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:13.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:58:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:14.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: Reconfiguring ceph-exporter.smithi086 (monmap changed)... 2024-02-19T18:58:14.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: Reconfiguring daemon ceph-exporter.smithi086 on smithi086 2024-02-19T18:58:14.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:14.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:14.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:58:14.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:14.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T18:58:14.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: Reconfiguring ceph-exporter.smithi086 (monmap changed)... 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: Reconfiguring daemon ceph-exporter.smithi086 on smithi086 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T18:58:15.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:16.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:15 smithi086 ceph-mon[37629]: Reconfiguring crash.smithi086 (monmap changed)... 2024-02-19T18:58:16.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:15 smithi086 ceph-mon[37629]: Reconfiguring daemon crash.smithi086 on smithi086 2024-02-19T18:58:16.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:15 smithi086 ceph-mon[37629]: Deploying daemon osd.0 on smithi039 2024-02-19T18:58:16.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:15 smithi086 ceph-mon[37629]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:16.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:16.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:16.077 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:15 smithi039 ceph-mon[28988]: Reconfiguring crash.smithi086 (monmap changed)... 2024-02-19T18:58:16.077 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:15 smithi039 ceph-mon[28988]: Reconfiguring daemon crash.smithi086 on smithi086 2024-02-19T18:58:16.077 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:15 smithi039 ceph-mon[28988]: Deploying daemon osd.0 on smithi039 2024-02-19T18:58:16.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:15 smithi039 ceph-mon[28988]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:16.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:16.078 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:16.955 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:16.955 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:16 smithi086 ceph-mon[37629]: Reconfiguring mgr.smithi086.huubbu (monmap changed)... 2024-02-19T18:58:16.955 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:58:16.955 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:58:16.955 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:16.955 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:16 smithi086 ceph-mon[37629]: Reconfiguring daemon mgr.smithi086.huubbu on smithi086 2024-02-19T18:58:17.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:17.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:16 smithi039 ceph-mon[28988]: Reconfiguring mgr.smithi086.huubbu (monmap changed)... 2024-02-19T18:58:17.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:58:17.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:58:17.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:17.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:16 smithi039 ceph-mon[28988]: Reconfiguring daemon mgr.smithi086.huubbu on smithi086 2024-02-19T18:58:17.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:17 smithi086 ceph-mon[37629]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:17.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:17 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:17.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:17 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:17.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:17 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:58:17.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:17 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:58:17.999 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:17 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:18.017 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:17 smithi039 ceph-mon[28988]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:18.017 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:17 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.017 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:17 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.017 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:17 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:58:18.018 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:17 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:58:18.018 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:17 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:18.815 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: Reconfiguring mon.smithi086 (monmap changed)... 2024-02-19T18:58:18.815 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: Reconfiguring daemon mon.smithi086 on smithi086 2024-02-19T18:58:18.815 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.815 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi039.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi039.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi039.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:18.816 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:18 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:19.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: Reconfiguring mon.smithi086 (monmap changed)... 2024-02-19T18:58:19.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: Reconfiguring daemon mon.smithi086 on smithi086 2024-02-19T18:58:19.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi039.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi039.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi039.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:19.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:18 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:20.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:58:20.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi039.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:58:20.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:20.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:58:20.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi039.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:58:20.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:58:20.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:19 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi039.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:58:20.161 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:58:20.162 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi039.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:58:20.162 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:20.162 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:58:20.162 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi039.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:58:20.162 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:58:20.162 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:19 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi039.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:58:20.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:21.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:21.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:21.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:21.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:21.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:21 smithi039 ceph-mon[28988]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:22.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:21 smithi086 ceph-mon[37629]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:23.117 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:22 smithi039 ceph-mon[28988]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:23.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:22 smithi086 ceph-mon[37629]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:24.183 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:23 smithi086 ceph-mon[37629]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T18:58:24.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:23 smithi039 ceph-mon[28988]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T18:58:24.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.276+0000 7f5e557fa700 1 -- 172.21.15.39:0/2668322514 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5e580611d0 con 0x7f5e4006c460 2024-02-19T18:58:24.295 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 0 on host 'smithi039' 2024-02-19T18:58:24.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e4006c460 msgr2=0x7f5e4006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:24.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e4006c460 0x7f5e4006e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00b5c0 tx=0x7f5e4c005960 comp rx=0 tx=0).stop 2024-02-19T18:58:24.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 msgr2=0x7f5e58196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:24.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 0x7f5e58196460 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800c8a0 tx=0x7f5e4800cc60 comp rx=0 tx=0).stop 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 shutdown_connections 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e4006c460 0x7f5e4006e920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e58101740 0x7f5e58196460 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 --2- 172.21.15.39:0/2668322514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e58104070 0x7f5e581969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 >> 172.21.15.39:0/2668322514 conn(0x7f5e580fb150 msgr2=0x7f5e580fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 shutdown_connections 2024-02-19T18:58:24.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:24.279+0000 7f5e5e86e700 1 -- 172.21.15.39:0/2668322514 wait complete. 2024-02-19T18:58:24.894 DEBUG:teuthology.orchestra.run.smithi039:osd.0> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.0.service 2024-02-19T18:58:24.899 INFO:tasks.cephadm:Deploying osd.1 on smithi039 with /dev/vg_nvme/lv_3... 2024-02-19T18:58:24.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: osdmap e7: 1 total, 0 up, 1 in 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:25.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-mon[28988]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:25.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[47506]: 2024-02-19T18:58:24.929+0000 7fb1c27de700 -1 osd.0 0 waiting for initial osdmap 2024-02-19T18:58:25.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[47506]: 2024-02-19T18:58:24.933+0000 7fb1bc5cf700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:58:25.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T18:58:25.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: osdmap e7: 1 total, 0 up, 1 in 2024-02-19T18:58:25.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:25.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T18:58:25.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:25.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:24 smithi086 ceph-mon[37629]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:25.682 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:58:26.196 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:25 smithi039 ceph-mon[28988]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-02-19T18:58:26.196 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:25 smithi039 ceph-mon[28988]: osdmap e8: 1 total, 0 up, 1 in 2024-02-19T18:58:26.196 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:25 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:26.196 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:25 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:26.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:25 smithi086 ceph-mon[37629]: from='osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-02-19T18:58:26.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:25 smithi086 ceph-mon[37629]: osdmap e8: 1 total, 0 up, 1 in 2024-02-19T18:58:26.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:25 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:26.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:25 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:27.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: purged_snaps scrub starts 2024-02-19T18:58:27.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: purged_snaps scrub ok 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] boot 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: osdmap e9: 1 total, 1 up, 1 in 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:26 smithi039 ceph-mon[28988]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:27.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: purged_snaps scrub starts 2024-02-19T18:58:27.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: purged_snaps scrub ok 2024-02-19T18:58:27.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:27.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: osd.0 [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] boot 2024-02-19T18:58:27.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: osdmap e9: 1 total, 1 up, 1 in 2024-02-19T18:58:27.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:58:27.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:27.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:27.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:26 smithi086 ceph-mon[37629]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:58:28.241 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T18:58:28.241 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T18:58:28.241 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-02-19T18:58:28.241 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-02-19T18:58:28.241 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0156823 s, 669 MB/s 2024-02-19T18:58:28.241 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-02-19T18:58:28.665 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:28 smithi039 ceph-mon[28988]: osdmap e10: 1 total, 1 up, 1 in 2024-02-19T18:58:29.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:28 smithi086 ceph-mon[37629]: osdmap e10: 1 total, 1 up, 1 in 2024-02-19T18:58:29.573 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_3 2024-02-19T18:58:29.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:29 smithi039 ceph-mon[28988]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:30.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:29 smithi086 ceph-mon[37629]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:58:30.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:31.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:31.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:31 smithi039 ceph-mon[28988]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:32.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:31 smithi086 ceph-mon[37629]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 -- 172.21.15.39:0/3478248222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478101750 msgr2=0x7fa478103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/3478248222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478101750 0x7fa478103b40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa468009910 tx=0x7fa468009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 -- 172.21.15.39:0/3478248222 shutdown_connections 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/3478248222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa478104080 0x7fa478106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/3478248222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478101750 0x7fa478103b40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 -- 172.21.15.39:0/3478248222 >> 172.21.15.39:0/3478248222 conn(0x7fa4780fb160 msgr2=0x7fa4780fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:32.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 -- 172.21.15.39:0/3478248222 shutdown_connections 2024-02-19T18:58:32.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 -- 172.21.15.39:0/3478248222 wait complete. 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.412+0000 7fa47d8bb700 1 Processor -- start 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa47d8bb700 1 -- start start 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa47d8bb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa478101750 0x7fa4781967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa47d8bb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 0x7fa478196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa47d8bb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa478197340 con 0x7fa478104080 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa47d8bb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa478197480 con 0x7fa478101750 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa4767fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 0x7fa478196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa4767fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 0x7fa478196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38734/0 (socket says 172.21.15.39:38734) 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa4767fc700 1 -- 172.21.15.39:0/363728344 learned_addr learned my addr 172.21.15.39:0/363728344 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa476ffd700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa478101750 0x7fa4781967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:32.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa4767fc700 1 -- 172.21.15.39:0/363728344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa478101750 msgr2=0x7fa4781967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa4767fc700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa478101750 0x7fa4781967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.413+0000 7fa4767fc700 1 -- 172.21.15.39:0/363728344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680095f0 con 0x7fa478104080 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.414+0000 7fa4767fc700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 0x7fa478196d20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00e920 tx=0x7fa46c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.414+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46c00c9d0 con 0x7fa478104080 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.414+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa46c010430 con 0x7fa478104080 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.414+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47819bf30 con 0x7fa478104080 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.414+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46c018650 con 0x7fa478104080 2024-02-19T18:58:32.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.414+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47819c450 con 0x7fa478104080 2024-02-19T18:58:32.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.415+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa478190630 con 0x7fa478104080 2024-02-19T18:58:32.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.415+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7fa46c00cb30 con 0x7fa478104080 2024-02-19T18:58:32.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.416+0000 7fa47c8b9700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa46006c460 0x7fa46006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:32.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.416+0000 7fa476ffd700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa46006c460 0x7fa46006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:32.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.416+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7fa46c014070 con 0x7fa478104080 2024-02-19T18:58:32.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.416+0000 7fa476ffd700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa46006c460 0x7fa46006e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa468009fd0 tx=0x7fa468005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:32.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.420+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa46c0520e0 con 0x7fa478104080 2024-02-19T18:58:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:32.628+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa47802d050 con 0x7fa46006c460 2024-02-19T18:58:33.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: Detected new or changed devices on smithi039 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi039 to 11840M 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:33.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:33 smithi086 ceph-mon[37629]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: Detected new or changed devices on smithi039 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi039 to 11840M 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:33.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='client.14276 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:33.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:58:33.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:58:33.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:33.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:33 smithi039 ceph-mon[28988]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:35.316 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:35 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:35.316 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:35 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:35.316 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:35 smithi039 ceph-mon[28988]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:35.453 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:35 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:35.453 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:35 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:35.454 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:35 smithi086 ceph-mon[37629]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:36.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:36 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1118422161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b6143b3-8bce-4be7-a199-57711f652a9a"}]: dispatch 2024-02-19T18:58:36.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:36 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1118422161' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b6143b3-8bce-4be7-a199-57711f652a9a"}]': finished 2024-02-19T18:58:36.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:36 smithi039 ceph-mon[28988]: osdmap e11: 2 total, 1 up, 2 in 2024-02-19T18:58:36.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:36 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:36.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:36 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1118422161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b6143b3-8bce-4be7-a199-57711f652a9a"}]: dispatch 2024-02-19T18:58:36.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:36 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1118422161' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b6143b3-8bce-4be7-a199-57711f652a9a"}]': finished 2024-02-19T18:58:36.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:36 smithi086 ceph-mon[37629]: osdmap e11: 2 total, 1 up, 2 in 2024-02-19T18:58:36.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:36 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:37.744 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:37 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3461183290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:37.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:37 smithi039 ceph-mon[28988]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:38.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:37 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3461183290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:38.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:37 smithi086 ceph-mon[37629]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:39.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:38 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:39.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:38 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:39.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:38 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:39.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:38 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:39.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:38 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:39.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:38 smithi086 ceph-mon[37629]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:39.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:39.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:39.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:39.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:39.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:38 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:39.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:38 smithi039 ceph-mon[28988]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:41.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:41 smithi039 ceph-mon[28988]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:42.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:41 smithi086 ceph-mon[37629]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:44.001 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:43 smithi039 ceph-mon[28988]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:44.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:43 smithi086 ceph-mon[37629]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:45.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T18:58:45.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:45.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:44 smithi039 ceph-mon[28988]: Deploying daemon osd.1 on smithi039 2024-02-19T18:58:45.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:44 smithi039 ceph-mon[28988]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T18:58:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:44 smithi086 ceph-mon[37629]: Deploying daemon osd.1 on smithi039 2024-02-19T18:58:45.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:44 smithi086 ceph-mon[37629]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:46.155 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:46.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:47.117 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:46 smithi039 ceph-mon[28988]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:47.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:46 smithi086 ceph-mon[37629]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:49.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:49 smithi039 ceph-mon[28988]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:49.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:49.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:49.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:50.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:49 smithi086 ceph-mon[37629]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:50.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:50.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:50.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:51.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:51 smithi039 ceph-mon[28988]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:51.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:51 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:51.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:51 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:52.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:51 smithi086 ceph-mon[37629]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:52.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:51 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:52.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:51 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:53.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.435+0000 7fa47c8b9700 1 -- 172.21.15.39:0/363728344 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa47802d050 con 0x7fa46006c460 2024-02-19T18:58:53.453 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 1 on host 'smithi039' 2024-02-19T18:58:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa46006c460 msgr2=0x7fa46006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa46006c460 0x7fa46006e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa468009fd0 tx=0x7fa468005940 comp rx=0 tx=0).stop 2024-02-19T18:58:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 msgr2=0x7fa478196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 0x7fa478196d20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00e920 tx=0x7fa46c00ec30 comp rx=0 tx=0).stop 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 shutdown_connections 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa46006c460 0x7fa46006e920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa478101750 0x7fa4781967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 --2- 172.21.15.39:0/363728344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa478104080 0x7fa478196d20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 >> 172.21.15.39:0/363728344 conn(0x7fa4780fb160 msgr2=0x7fa4780fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 shutdown_connections 2024-02-19T18:58:53.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:58:53.438+0000 7fa47d8bb700 1 -- 172.21.15.39:0/363728344 wait complete. 2024-02-19T18:58:54.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:53 smithi086 ceph-mon[37629]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:54.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:53 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:54.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:53 smithi086 ceph-mon[37629]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T18:58:54.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:53 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:54.040 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:53 smithi039 ceph-mon[28988]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:54.040 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:53 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:54.040 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:53 smithi039 ceph-mon[28988]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T18:58:54.040 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:53 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:54.063 DEBUG:teuthology.orchestra.run.smithi039:osd.1> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.1.service 2024-02-19T18:58:54.065 INFO:tasks.cephadm:Deploying osd.2 on smithi039 with /dev/vg_nvme/lv_2... 2024-02-19T18:58:54.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T18:58:54.782 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:58:55.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[54808]: 2024-02-19T18:58:55.429+0000 7f510e8a5700 -1 osd.1 0 waiting for initial osdmap 2024-02-19T18:58:55.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-mon[28988]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T18:58:55.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-mon[28988]: osdmap e12: 2 total, 1 up, 2 in 2024-02-19T18:58:55.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:55.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-mon[28988]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T18:58:55.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-mon[28988]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:55.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[54808]: 2024-02-19T18:58:55.437+0000 7f5107e95700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:58:55.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:55 smithi086 ceph-mon[37629]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T18:58:55.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:55 smithi086 ceph-mon[37629]: osdmap e12: 2 total, 1 up, 2 in 2024-02-19T18:58:55.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:55.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:55 smithi086 ceph-mon[37629]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T18:58:55.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:55 smithi086 ceph-mon[37629]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: purged_snaps scrub starts 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: purged_snaps scrub ok 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: osdmap e13: 2 total, 1 up, 2 in 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:56.743 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:56 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:56.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: purged_snaps scrub starts 2024-02-19T18:58:56.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: purged_snaps scrub ok 2024-02-19T18:58:56.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: from='osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-02-19T18:58:56.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: osdmap e13: 2 total, 1 up, 2 in 2024-02-19T18:58:56.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:56.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:56.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:56 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134521 s, 779 MB/s 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 2024-02-19T18:58:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-02-19T18:58:57.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:57 smithi039 ceph-mon[28988]: osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] boot 2024-02-19T18:58:57.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:57 smithi039 ceph-mon[28988]: osdmap e14: 2 total, 2 up, 2 in 2024-02-19T18:58:57.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:57 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:57.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:57 smithi039 ceph-mon[28988]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:57 smithi086 ceph-mon[37629]: osd.1 [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] boot 2024-02-19T18:58:57.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:57 smithi086 ceph-mon[37629]: osdmap e14: 2 total, 2 up, 2 in 2024-02-19T18:58:57.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:57 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:58:57.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:57 smithi086 ceph-mon[37629]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:58:58.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_2 2024-02-19T18:58:58.575 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:58 smithi039 ceph-mon[28988]: osdmap e15: 2 total, 2 up, 2 in 2024-02-19T18:58:58.575 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:58 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:58.575 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:58 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:58.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:58 smithi086 ceph-mon[37629]: osdmap e15: 2 total, 2 up, 2 in 2024-02-19T18:58:58.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:58 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:58.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:58 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:58:59.425 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T18:58:59.667 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:58:59 smithi039 ceph-mon[28988]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:58:59.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:58:59 smithi086 ceph-mon[37629]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:00.990 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:01.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:02.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:01 smithi086 ceph-mon[37629]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:02.071 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:01 smithi039 ceph-mon[28988]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:02.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 -- 172.21.15.39:0/2002685363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 msgr2=0x7faa6c106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 --2- 172.21.15.39:0/2002685363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c106460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa60009910 tx=0x7faa60009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:02.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 -- 172.21.15.39:0/2002685363 shutdown_connections 2024-02-19T18:59:02.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 --2- 172.21.15.39:0/2002685363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c106460 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 --2- 172.21.15.39:0/2002685363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa6c101740 0x7faa6c103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 -- 172.21.15.39:0/2002685363 >> 172.21.15.39:0/2002685363 conn(0x7faa6c0fb150 msgr2=0x7faa6c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:02.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 -- 172.21.15.39:0/2002685363 shutdown_connections 2024-02-19T18:59:02.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.345+0000 7faa71fec700 1 -- 172.21.15.39:0/2002685363 wait complete. 2024-02-19T18:59:02.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa71fec700 1 Processor -- start 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa71fec700 1 -- start start 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa71fec700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa6c101740 0x7faa6c196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa71fec700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa71fec700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c196fc0 con 0x7faa6c104070 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa71fec700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c197100 con 0x7faa6c101740 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa6affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa6affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c1969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52228/0 (socket says 172.21.15.39:52228) 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa6affd700 1 -- 172.21.15.39:0/974273464 learned_addr learned my addr 172.21.15.39:0/974273464 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.346+0000 7faa6b7fe700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa6c101740 0x7faa6c196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa6affd700 1 -- 172.21.15.39:0/974273464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa6c101740 msgr2=0x7faa6c196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa6affd700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa6c101740 0x7faa6c196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa6affd700 1 -- 172.21.15.39:0/974273464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600095f0 con 0x7faa6c104070 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa6affd700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c1969a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faa60006010 tx=0x7faa6000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa6001d070 con 0x7faa6c104070 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faa60005dc0 con 0x7faa6c104070 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c19bb50 con 0x7faa6c104070 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa60021410 con 0x7faa6c104070 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.347+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c19c040 con 0x7faa6c104070 2024-02-19T18:59:02.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.348+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c190620 con 0x7faa6c104070 2024-02-19T18:59:02.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.349+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7faa6002b440 con 0x7faa6c104070 2024-02-19T18:59:02.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.349+0000 7faa68ff9700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa5406c470 0x7faa5406e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.349+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7faa6008bcc0 con 0x7faa6c104070 2024-02-19T18:59:02.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.349+0000 7faa6b7fe700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa5406c470 0x7faa5406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.350+0000 7faa6b7fe700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa5406c470 0x7faa5406e930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa5c005950 tx=0x7faa5c00b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:02.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.354+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faa6005ac80 con 0x7faa6c104070 2024-02-19T18:59:02.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:02.563+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faa6c0611d0 con 0x7faa5406c470 2024-02-19T18:59:03.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: Detected new or changed devices on smithi039 2024-02-19T18:59:03.196 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi039 to 5920M 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:03.197 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:03 smithi086 ceph-mon[37629]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: Detected new or changed devices on smithi039 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi039 to 5920M 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:03.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:03.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:59:03.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:59:03.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:03.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:03 smithi039 ceph-mon[28988]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:05.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:05 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:05.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:05 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:05.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:05 smithi039 ceph-mon[28988]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:05.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:05 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:05.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:05 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:05.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:05 smithi086 ceph-mon[37629]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:06.472 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:06 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1321608637' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcb75e1b-3f22-4e89-9bdd-ce8e833a3770"}]: dispatch 2024-02-19T18:59:06.472 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:06 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1321608637' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcb75e1b-3f22-4e89-9bdd-ce8e833a3770"}]': finished 2024-02-19T18:59:06.472 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:06 smithi086 ceph-mon[37629]: osdmap e16: 3 total, 2 up, 3 in 2024-02-19T18:59:06.472 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:06 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:06.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:06 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1321608637' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcb75e1b-3f22-4e89-9bdd-ce8e833a3770"}]: dispatch 2024-02-19T18:59:06.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:06 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1321608637' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcb75e1b-3f22-4e89-9bdd-ce8e833a3770"}]': finished 2024-02-19T18:59:06.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:06 smithi039 ceph-mon[28988]: osdmap e16: 3 total, 2 up, 3 in 2024-02-19T18:59:06.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:06 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:07.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:07 smithi039 ceph-mon[28988]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:07.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:07 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1514696749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:59:08.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:07 smithi086 ceph-mon[37629]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:08.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:07 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1514696749' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:59:09.044 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:09 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:09.044 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:09 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:09.044 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:09 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:09.044 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:09 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:09.044 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:09 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:09.044 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:09 smithi039 ceph-mon[28988]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:09.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:09 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:09.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:09 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:09.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:09 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:09.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:09 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:09.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:09 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:09.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:09 smithi086 ceph-mon[37629]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:11.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:11 smithi039 ceph-mon[28988]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:12.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:11 smithi086 ceph-mon[37629]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:14.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:13 smithi086 ceph-mon[37629]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:14.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:13 smithi039 ceph-mon[28988]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:15.206 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T18:59:15.206 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:15.206 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:14 smithi039 ceph-mon[28988]: Deploying daemon osd.2 on smithi039 2024-02-19T18:59:15.206 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:14 smithi039 ceph-mon[28988]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:15.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T18:59:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:14 smithi086 ceph-mon[37629]: Deploying daemon osd.2 on smithi039 2024-02-19T18:59:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:14 smithi086 ceph-mon[37629]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:16.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:16.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:17.169 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:16 smithi039 ceph-mon[28988]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:17.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:16 smithi086 ceph-mon[37629]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:19.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:19 smithi039 ceph-mon[28988]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:20.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:19 smithi086 ceph-mon[37629]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:20 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:21.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:21.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:21.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:20 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:21.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:21 smithi039 ceph-mon[28988]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:22.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:21 smithi086 ceph-mon[37629]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:23.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:23 smithi039 ceph-mon[28988]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:23.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:23 smithi039 ceph-mon[28988]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T18:59:23.989 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 2 on host 'smithi039' 2024-02-19T18:59:23.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.970+0000 7faa68ff9700 1 -- 172.21.15.39:0/974273464 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faa6c0611d0 con 0x7faa5406c470 2024-02-19T18:59:23.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.972+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa5406c470 msgr2=0x7faa5406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa5406c470 0x7faa5406e930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa5c005950 tx=0x7faa5c00b410 comp rx=0 tx=0).stop 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 msgr2=0x7faa6c1969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c1969a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faa60006010 tx=0x7faa6000bd70 comp rx=0 tx=0).stop 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 shutdown_connections 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7faa5406c470 0x7faa5406e930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa6c101740 0x7faa6c196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 --2- 172.21.15.39:0/974273464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c104070 0x7faa6c1969a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 >> 172.21.15.39:0/974273464 conn(0x7faa6c0fb150 msgr2=0x7faa6c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 shutdown_connections 2024-02-19T18:59:23.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T18:59:23.973+0000 7faa71fec700 1 -- 172.21.15.39:0/974273464 wait complete. 2024-02-19T18:59:24.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:23 smithi086 ceph-mon[37629]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:24.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:23 smithi086 ceph-mon[37629]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T18:59:24.609 DEBUG:teuthology.orchestra.run.smithi039:osd.2> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.2.service 2024-02-19T18:59:24.612 INFO:tasks.cephadm:Deploying osd.3 on smithi086 with /dev/vg_nvme/lv_4... 2024-02-19T18:59:24.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T18:59:24.823 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T18:59:24.922 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[62484]: 2024-02-19T18:59:24.810+0000 7f5c66d56700 -1 osd.2 0 waiting for initial osdmap 2024-02-19T18:59:24.922 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[62484]: 2024-02-19T18:59:24.818+0000 7f5c5fb45700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: osdmap e17: 3 total, 2 up, 3 in 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: osdmap e18: 3 total, 2 up, 3 in 2024-02-19T18:59:24.923 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: osdmap e17: 3 total, 2 up, 3 in 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: osdmap e18: 3 total, 2 up, 3 in 2024-02-19T18:59:24.941 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:26.135 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:25 smithi086 ceph-mon[37629]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:26.135 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:25 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:26.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:25 smithi039 ceph-mon[28988]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:26.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:25 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131155 s, 799 MB/s 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 2024-02-19T18:59:26.643 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-02-19T18:59:27.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:26 smithi039 ceph-mon[28988]: purged_snaps scrub starts 2024-02-19T18:59:27.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:26 smithi039 ceph-mon[28988]: purged_snaps scrub ok 2024-02-19T18:59:27.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:27.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:26 smithi039 ceph-mon[28988]: osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] boot 2024-02-19T18:59:27.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:26 smithi039 ceph-mon[28988]: osdmap e19: 3 total, 3 up, 3 in 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:26 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:26 smithi086 ceph-mon[37629]: purged_snaps scrub starts 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:26 smithi086 ceph-mon[37629]: purged_snaps scrub ok 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:26 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:26 smithi086 ceph-mon[37629]: osd.2 [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] boot 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:26 smithi086 ceph-mon[37629]: osdmap e19: 3 total, 3 up, 3 in 2024-02-19T18:59:27.269 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:26 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:59:27.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-02-19T18:59:27.493 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T18:59:28.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:28.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:59:28.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-19T18:59:28.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: osdmap e20: 3 total, 3 up, 3 in 2024-02-19T18:59:28.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:28.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:59:28.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:27 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: osdmap e20: 3 total, 3 up, 3 in 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:59:28.267 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:27 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:28.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:28 smithi039 sudo[64352]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:59:28.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:28 smithi039 sudo[64352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:28.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:28 smithi039 sudo[64352]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:28.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:28 smithi039 sudo[64352]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 -- 172.21.15.86:0/4204322991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 msgr2=0x7f4b14103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 --2- 172.21.15.86:0/4204322991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14103b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009910 tx=0x7f4b04009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 -- 172.21.15.86:0/4204322991 shutdown_connections 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 --2- 172.21.15.86:0/4204322991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b141040c0 0x7f4b141064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 --2- 172.21.15.86:0/4204322991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14103b80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 -- 172.21.15.86:0/4204322991 >> 172.21.15.86:0/4204322991 conn(0x7f4b140fb120 msgr2=0x7f4b140fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.063+0000 7f4b19a42700 1 -- 172.21.15.86:0/4204322991 shutdown_connections 2024-02-19T18:59:29.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.064+0000 7f4b19a42700 1 -- 172.21.15.86:0/4204322991 wait complete. 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.064+0000 7f4b19a42700 1 Processor -- start 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.064+0000 7f4b19a42700 1 -- start start 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b19a42700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b19a42700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b141040c0 0x7f4b141969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b19a42700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14196fd0 con 0x7f4b141040c0 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b19a42700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14197110 con 0x7f4b14101790 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58480/0 (socket says 172.21.15.86:58480) 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 -- 172.21.15.86:0/1964694650 learned_addr learned my addr 172.21.15.86:0/1964694650 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b127fc700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b141040c0 0x7f4b141969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 -- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b141040c0 msgr2=0x7f4b141969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b141040c0 0x7f4b141969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 -- 172.21.15.86:0/1964694650 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040095f0 con 0x7f4b14101790 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b12ffd700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14196470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4b04005b40 tx=0x7f4b0400bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0401d070 con 0x7f4b14101790 2024-02-19T18:59:29.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1419bb60 con 0x7f4b14101790 2024-02-19T18:59:29.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4b0400f460 con 0x7f4b14101790 2024-02-19T18:59:29.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b04021620 con 0x7f4b14101790 2024-02-19T18:59:29.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.065+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1419c050 con 0x7f4b14101790 2024-02-19T18:59:29.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.066+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b14190610 con 0x7f4b14101790 2024-02-19T18:59:29.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.067+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f4b0400fa40 con 0x7f4b14101790 2024-02-19T18:59:29.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.067+0000 7f4b18a40700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4afc06c470 0x7f4afc06e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:29.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.067+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3232+0+0 (secure 0 0 0) 0x7f4b0408c030 con 0x7f4b14101790 2024-02-19T18:59:29.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.067+0000 7f4b127fc700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4afc06c470 0x7f4afc06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:29.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.068+0000 7f4b127fc700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4afc06c470 0x7f4afc06e930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b08005950 tx=0x7f4b0800b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:29.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.072+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4b0405ad60 con 0x7f4b14101790 2024-02-19T18:59:29.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:29 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-19T18:59:29.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:29 smithi086 ceph-mon[37629]: osdmap e21: 3 total, 3 up, 3 in 2024-02-19T18:59:29.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:29 smithi086 ceph-mon[37629]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:29.267+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b140611d0 con 0x7f4afc06c470 2024-02-19T18:59:29.422 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:28 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-19T18:59:29.422 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:28 smithi039 ceph-mon[28988]: osdmap e21: 3 total, 3 up, 3 in 2024-02-19T18:59:29.422 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:28 smithi039 ceph-mon[28988]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:29.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[64363]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:59:29.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[64363]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:29.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[64363]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:29.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[64363]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:29.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[65911]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:59:29.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[65911]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:29.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[65911]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:29.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[65911]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:29.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67103]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:59:29.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:29.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67103]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:29.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67103]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:30.147+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4b040576a0 con 0x7f4b14101790 2024-02-19T18:59:30.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 ceph-mon[28988]: osdmap e22: 3 total, 3 up, 3 in 2024-02-19T18:59:30.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:30.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 ceph-mon[28988]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:30.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:59:30.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:59:30.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:30.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67107]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:59:30.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67107]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:30.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67107]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:30.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:29 smithi039 sudo[67107]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:30.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:30 smithi086 ceph-mon[37629]: osdmap e22: 3 total, 3 up, 3 in 2024-02-19T18:59:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:30 smithi086 ceph-mon[37629]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:59:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:59:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:30.493 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67110]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:59:30.493 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:30.493 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67110]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:30.493 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67110]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:30.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67113]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:59:30.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:30.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67113]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:30.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67113]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:31.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67116]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:59:31.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:31.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67116]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:31.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:30 smithi039 sudo[67116]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:31.275 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41317]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:59:31.275 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:31.275 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41317]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:31.275 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41317]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41322]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41322]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41322]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 sudo[41322]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: mgrmap e19: smithi039.ydmpuh(active, since 2m), standbys: smithi086.huubbu 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: Detected new or changed devices on smithi039 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:59:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: mgrmap e19: smithi039.ydmpuh(active, since 2m), standbys: smithi086.huubbu 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: Detected new or changed devices on smithi039 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:31.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T18:59:31.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T18:59:32.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:59:32.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:59:32.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:32.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi039 to 3946M 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/77859741' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3353e0-09f3-4582-af41-33ed9e1581c8"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3353e0-09f3-4582-af41-33ed9e1581c8"}]: dispatch 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c3353e0-09f3-4582-af41-33ed9e1581c8"}]': finished 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: osdmap e23: 4 total, 3 up, 4 in 2024-02-19T18:59:32.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:32.532 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi039 to 3946M 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/77859741' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3353e0-09f3-4582-af41-33ed9e1581c8"}]: dispatch 2024-02-19T18:59:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c3353e0-09f3-4582-af41-33ed9e1581c8"}]: dispatch 2024-02-19T18:59:32.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c3353e0-09f3-4582-af41-33ed9e1581c8"}]': finished 2024-02-19T18:59:32.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: osdmap e23: 4 total, 3 up, 4 in 2024-02-19T18:59:32.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:33.563 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:33 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/1588905822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:59:33.563 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:33 smithi039 ceph-mon[28988]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:33.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:33 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/1588905822' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:59:33.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:33 smithi086 ceph-mon[37629]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:35.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:34 smithi039 ceph-mon[28988]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:35.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:34 smithi086 ceph-mon[37629]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:36.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:36 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:36.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:36 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:36.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:36 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:36.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:36 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:36.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:36 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:36 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:36 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:36.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:36 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:36 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:36.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:36 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:37.782 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:37 smithi086 ceph-mon[37629]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:37.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:37 smithi039 ceph-mon[28988]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:39.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:39 smithi086 ceph-mon[37629]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:39.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T18:59:39.884 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:40.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:39 smithi039 ceph-mon[28988]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:40.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T18:59:40.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:41.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:40 smithi086 ceph-mon[37629]: Deploying daemon osd.3 on smithi086 2024-02-19T18:59:41.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:40 smithi039 ceph-mon[28988]: Deploying daemon osd.3 on smithi086 2024-02-19T18:59:41.909 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:41 smithi086 ceph-mon[37629]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:42.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:41 smithi039 ceph-mon[28988]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:44.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:43 smithi086 ceph-mon[37629]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:44.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:43 smithi039 ceph-mon[28988]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:45.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:45.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:45.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:45.602 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:45 smithi086 ceph-mon[37629]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:45.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:45.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:45.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:45.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:45 smithi039 ceph-mon[28988]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:46.596 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:46 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:46.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:46 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:47.737 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:47 smithi086 ceph-mon[37629]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:47.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:47 smithi039 ceph-mon[28988]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:47.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.777+0000 7f4b18a40700 1 -- 172.21.15.86:0/1964694650 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4b140611d0 con 0x7f4afc06c470 2024-02-19T18:59:47.794 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 3 on host 'smithi086' 2024-02-19T18:59:47.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4afc06c470 msgr2=0x7f4afc06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:47.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4afc06c470 0x7f4afc06e930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b08005950 tx=0x7f4b0800b410 comp rx=0 tx=0).stop 2024-02-19T18:59:47.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 msgr2=0x7f4b14196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:47.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14196470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4b04005b40 tx=0x7f4b0400bd70 comp rx=0 tx=0).stop 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 shutdown_connections 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4afc06c470 0x7f4afc06e930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b14101790 0x7f4b14196470 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 --2- 172.21.15.86:0/1964694650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b141040c0 0x7f4b141969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 >> 172.21.15.86:0/1964694650 conn(0x7f4b140fb120 msgr2=0x7f4b140fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 shutdown_connections 2024-02-19T18:59:47.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:47.778+0000 7f4b19a42700 1 -- 172.21.15.86:0/1964694650 wait complete. 2024-02-19T18:59:48.453 DEBUG:teuthology.orchestra.run.smithi086:osd.3> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.3.service 2024-02-19T18:59:48.456 INFO:tasks.cephadm:Deploying osd.4 on smithi086 with /dev/vg_nvme/lv_3... 2024-02-19T18:59:48.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T18:59:48.887 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T18:59:49.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:48 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:49.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:48 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:49.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:48 smithi086 ceph-mon[37629]: from='osd.3 [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:59:49.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:48 smithi086 ceph-mon[37629]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:59:49.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:48 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:49.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:48 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:49.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:48 smithi039 ceph-mon[28988]: from='osd.3 [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:59:49.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:48 smithi039 ceph-mon[28988]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:59:50.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: osdmap e24: 4 total, 3 up, 4 in 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: from='osd.3 [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:50.073 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:50.073 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[43584]: 2024-02-19T18:59:49.778+0000 7f63d02fb700 -1 osd.3 0 waiting for initial osdmap 2024-02-19T18:59:50.073 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 18:59:49 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[43584]: 2024-02-19T18:59:49.785+0000 7f63ca8ed700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: osdmap e24: 4 total, 3 up, 4 in 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: from='osd.3 [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:50.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: osdmap e25: 4 total, 3 up, 4 in 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: osd.3 [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] boot 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: osdmap e26: 4 total, 4 up, 4 in 2024-02-19T18:59:50.967 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:50.968 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T18:59:50.968 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T18:59:50.968 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-02-19T18:59:50.968 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-02-19T18:59:50.968 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126745 s, 827 MB/s 2024-02-19T18:59:50.968 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-02-19T18:59:51.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-02-19T18:59:51.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: osdmap e25: 4 total, 3 up, 4 in 2024-02-19T18:59:51.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:51.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:51.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: osd.3 [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] boot 2024-02-19T18:59:51.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: osdmap e26: 4 total, 4 up, 4 in 2024-02-19T18:59:51.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:59:51.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-02-19T18:59:52.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:51 smithi086 ceph-mon[37629]: purged_snaps scrub starts 2024-02-19T18:59:52.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:51 smithi086 ceph-mon[37629]: purged_snaps scrub ok 2024-02-19T18:59:52.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:51 smithi086 ceph-mon[37629]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:52.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:51 smithi086 ceph-mon[37629]: osdmap e27: 4 total, 4 up, 4 in 2024-02-19T18:59:52.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:51 smithi039 ceph-mon[28988]: purged_snaps scrub starts 2024-02-19T18:59:52.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:51 smithi039 ceph-mon[28988]: purged_snaps scrub ok 2024-02-19T18:59:52.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:51 smithi039 ceph-mon[28988]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:59:52.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:51 smithi039 ceph-mon[28988]: osdmap e27: 4 total, 4 up, 4 in 2024-02-19T18:59:52.677 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T18:59:53.973 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:53 smithi086 ceph-mon[37629]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:59:53.973 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:53 smithi086 ceph-mon[37629]: osdmap e28: 4 total, 4 up, 4 in 2024-02-19T18:59:54.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:53 smithi039 ceph-mon[28988]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:59:54.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:53 smithi039 ceph-mon[28988]: osdmap e28: 4 total, 4 up, 4 in 2024-02-19T18:59:54.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2083689556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 msgr2=0x7ff5b8101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:54.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2083689556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b8101220 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009910 tx=0x7ff5b4009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:54.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2083689556 shutdown_connections 2024-02-19T18:59:54.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2083689556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b8101220 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:54.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2083689556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80ffbc0 0x7ff5b80fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2083689556 >> 172.21.15.86:0/2083689556 conn(0x7ff5b80fb160 msgr2=0x7ff5b80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2083689556 shutdown_connections 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.886+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2083689556 wait complete. 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5c00f4700 1 Processor -- start 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5c00f4700 1 -- start start 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5c00f4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80ffbc0 0x7ff5b8069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5c00f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b819b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5c00f4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b819bad0 con 0x7ff5b80ffbc0 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5c00f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b819bc10 con 0x7ff5b8100dc0 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5bd68f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b819b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5bd68f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b819b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48056/0 (socket says 172.21.15.86:48056) 2024-02-19T18:59:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5bd68f700 1 -- 172.21.15.86:0/2204539023 learned_addr learned my addr 172.21.15.86:0/2204539023 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.887+0000 7ff5bde90700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80ffbc0 0x7ff5b8069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5bd68f700 1 -- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80ffbc0 msgr2=0x7ff5b8069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5bd68f700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80ffbc0 0x7ff5b8069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5bd68f700 1 -- 172.21.15.86:0/2204539023 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40095f0 con 0x7ff5b8100dc0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5bd68f700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b819b500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff5b400bda0 tx=0x7ff5b400bdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5b401d070 con 0x7ff5b8100dc0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff5b400f460 con 0x7ff5b8100dc0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b819be90 con 0x7ff5b8100dc0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5b4021640 con 0x7ff5b8100dc0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.888+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b819c350 con 0x7ff5b8100dc0 2024-02-19T18:59:54.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.889+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b8068af0 con 0x7ff5b8100dc0 2024-02-19T18:59:54.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.889+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff5b400f5d0 con 0x7ff5b8100dc0 2024-02-19T18:59:54.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.890+0000 7ff5aaffd700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff5a406c4e0 0x7ff5a406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:54.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.890+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7ff5b408c480 con 0x7ff5b8100dc0 2024-02-19T18:59:54.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.890+0000 7ff5bde90700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff5a406c4e0 0x7ff5a406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:54.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.890+0000 7ff5bde90700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff5a406c4e0 0x7ff5a406e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8100c20 tx=0x7ff5ac00a380 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:54.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:54.894+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff5b405ae10 con 0x7ff5b8100dc0 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: osdmap e29: 4 total, 4 up, 4 in 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:54 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T18:59:55.091+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5b8105830 con 0x7ff5a406c4e0 2024-02-19T18:59:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: osdmap e29: 4 total, 4 up, 4 in 2024-02-19T18:59:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:59:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:54 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T18:59:56.031 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: Detected new or changed devices on smithi086 2024-02-19T18:59:56.031 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi086 to 14912M 2024-02-19T18:59:56.031 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:59:56.031 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: osdmap e30: 4 total, 4 up, 4 in 2024-02-19T18:59:56.031 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:59:56.031 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:59:56.032 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:55 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: Detected new or changed devices on smithi086 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi086 to 14912M 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: osdmap e30: 4 total, 4 up, 4 in 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:59:56.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:55 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:56.936 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:56 smithi086 ceph-mon[37629]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:57.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:56 smithi039 ceph-mon[28988]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:58.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:57 smithi086 ceph-mon[37629]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:59:58.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:57 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/3198662919' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e52e70-7bbd-4395-98b1-2bc8eabc2fb6"}]: dispatch 2024-02-19T18:59:58.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:57 smithi086 ceph-mon[37629]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e52e70-7bbd-4395-98b1-2bc8eabc2fb6"}]: dispatch 2024-02-19T18:59:58.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:57 smithi086 ceph-mon[37629]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62e52e70-7bbd-4395-98b1-2bc8eabc2fb6"}]': finished 2024-02-19T18:59:58.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:57 smithi086 ceph-mon[37629]: osdmap e31: 5 total, 4 up, 5 in 2024-02-19T18:59:58.099 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:57 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:59:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:57 smithi039 ceph-mon[28988]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:59:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:57 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3198662919' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e52e70-7bbd-4395-98b1-2bc8eabc2fb6"}]: dispatch 2024-02-19T18:59:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:57 smithi039 ceph-mon[28988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62e52e70-7bbd-4395-98b1-2bc8eabc2fb6"}]: dispatch 2024-02-19T18:59:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:57 smithi039 ceph-mon[28988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62e52e70-7bbd-4395-98b1-2bc8eabc2fb6"}]': finished 2024-02-19T18:59:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:57 smithi039 ceph-mon[28988]: osdmap e31: 5 total, 4 up, 5 in 2024-02-19T18:59:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:57 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:59:59.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:58 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3722191366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:59:59.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:58 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/3722191366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T19:00:00.237 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 18:59:59 smithi086 ceph-mon[37629]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-19T19:00:00.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 18:59:59 smithi039 ceph-mon[28988]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-19T19:00:01.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:00 smithi039 ceph-mon[28988]: overall HEALTH_OK 2024-02-19T19:00:01.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:01.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:00 smithi086 ceph-mon[37629]: overall HEALTH_OK 2024-02-19T19:00:01.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:01.943 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:01 smithi086 ceph-mon[37629]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-02-19T19:00:02.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:01 smithi039 ceph-mon[28988]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-02-19T19:00:04.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:03 smithi086 ceph-mon[37629]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-19T19:00:04.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:03 smithi039 ceph-mon[28988]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-19T19:00:05.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:04 smithi039 ceph-mon[28988]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-19T19:00:05.282 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:04 smithi086 ceph-mon[37629]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-19T19:00:06.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:05 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:00:06.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:05 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:06.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:05 smithi086 ceph-mon[37629]: Deploying daemon osd.4 on smithi086 2024-02-19T19:00:06.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:05 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:00:06.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:05 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:06.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:05 smithi039 ceph-mon[28988]: Deploying daemon osd.4 on smithi086 2024-02-19T19:00:07.225 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:06 smithi086 ceph-mon[37629]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-19T19:00:07.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:06 smithi039 ceph-mon[28988]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-19T19:00:09.974 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:09 smithi086 ceph-mon[37629]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:10.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:09 smithi039 ceph-mon[28988]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:11.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:11.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:11.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:11 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:11.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:11 smithi039 ceph-mon[28988]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:11.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:11 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:11.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:11 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:11.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:11 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:11.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:11 smithi086 ceph-mon[37629]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:13.951 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:13 smithi086 ceph-mon[37629]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:14.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:13 smithi039 ceph-mon[28988]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:14.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.307+0000 7ff5aaffd700 1 -- 172.21.15.86:0/2204539023 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5b8105830 con 0x7ff5a406c4e0 2024-02-19T19:00:14.325 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 4 on host 'smithi086' 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.309+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff5a406c4e0 msgr2=0x7ff5a406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.309+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff5a406c4e0 0x7ff5a406e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8100c20 tx=0x7ff5ac00a380 comp rx=0 tx=0).stop 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.309+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 msgr2=0x7ff5b819b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.309+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b819b500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff5b400bda0 tx=0x7ff5b400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 shutdown_connections 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff5a406c4e0 0x7ff5a406e9a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5b80ffbc0 0x7ff5b8069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:14.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 --2- 172.21.15.86:0/2204539023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5b8100dc0 0x7ff5b819b500 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:14.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 >> 172.21.15.86:0/2204539023 conn(0x7ff5b80fb160 msgr2=0x7ff5b81040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:14.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 shutdown_connections 2024-02-19T19:00:14.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:14.310+0000 7ff5c00f4700 1 -- 172.21.15.86:0/2204539023 wait complete. 2024-02-19T19:00:14.917 DEBUG:teuthology.orchestra.run.smithi086:osd.4> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.4.service 2024-02-19T19:00:14.919 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_2... 2024-02-19T19:00:14.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T19:00:15.173 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T19:00:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:14 smithi039 ceph-mon[28988]: from='osd.4 [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:00:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:14 smithi039 ceph-mon[28988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:00:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:14 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:14 smithi039 ceph-mon[28988]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:14 smithi086 ceph-mon[37629]: from='osd.4 [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:00:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:14 smithi086 ceph-mon[37629]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:00:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:14 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:14 smithi086 ceph-mon[37629]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: osdmap e32: 5 total, 4 up, 5 in 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='osd.4 [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:16.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:16.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: osdmap e32: 5 total, 4 up, 5 in 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='osd.4 [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:16.284 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[50055]: 2024-02-19T19:00:15.981+0000 7f5ea5c9a700 -1 osd.4 0 waiting for initial osdmap 2024-02-19T19:00:16.284 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:00:15 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[50055]: 2024-02-19T19:00:15.990+0000 7f5e9ca85700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:00:17.184 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:16 smithi086 ceph-mon[37629]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-02-19T19:00:17.184 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:16 smithi086 ceph-mon[37629]: osdmap e33: 5 total, 4 up, 5 in 2024-02-19T19:00:17.184 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:17.185 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:17.185 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:16 smithi086 ceph-mon[37629]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:17.185 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:16 smithi039 ceph-mon[28988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-02-19T19:00:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:16 smithi039 ceph-mon[28988]: osdmap e33: 5 total, 4 up, 5 in 2024-02-19T19:00:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:16 smithi039 ceph-mon[28988]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T19:00:17.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T19:00:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T19:00:17.290 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-02-19T19:00:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-02-19T19:00:17.290 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141763 s, 740 MB/s 2024-02-19T19:00:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-02-19T19:00:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:17 smithi086 ceph-mon[37629]: purged_snaps scrub starts 2024-02-19T19:00:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:17 smithi086 ceph-mon[37629]: purged_snaps scrub ok 2024-02-19T19:00:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:17 smithi086 ceph-mon[37629]: osd.4 [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] boot 2024-02-19T19:00:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:17 smithi086 ceph-mon[37629]: osdmap e34: 5 total, 5 up, 5 in 2024-02-19T19:00:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:17 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:18.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:17 smithi039 ceph-mon[28988]: purged_snaps scrub starts 2024-02-19T19:00:18.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:17 smithi039 ceph-mon[28988]: purged_snaps scrub ok 2024-02-19T19:00:18.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:17 smithi039 ceph-mon[28988]: osd.4 [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] boot 2024-02-19T19:00:18.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:17 smithi039 ceph-mon[28988]: osdmap e34: 5 total, 5 up, 5 in 2024-02-19T19:00:18.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:17 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:18.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-02-19T19:00:19.047 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T19:00:19.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:19 smithi086 ceph-mon[37629]: osdmap e35: 5 total, 5 up, 5 in 2024-02-19T19:00:19.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:19 smithi086 ceph-mon[37629]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:19.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:19 smithi039 ceph-mon[28988]: osdmap e35: 5 total, 5 up, 5 in 2024-02-19T19:00:19.495 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:19 smithi039 ceph-mon[28988]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:21.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.119+0000 7fd62b323700 1 -- 172.21.15.86:0/2736171538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd624100da0 msgr2=0x7fd624101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:21.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.119+0000 7fd62b323700 1 --2- 172.21.15.86:0/2736171538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd624100da0 0x7fd624101200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd614009910 tx=0x7fd614009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:21.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.119+0000 7fd62b323700 1 -- 172.21.15.86:0/2736171538 shutdown_connections 2024-02-19T19:00:21.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.119+0000 7fd62b323700 1 --2- 172.21.15.86:0/2736171538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd624100da0 0x7fd624101200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.119+0000 7fd62b323700 1 --2- 172.21.15.86:0/2736171538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6240ffba0 0x7fd6240fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.119+0000 7fd62b323700 1 -- 172.21.15.86:0/2736171538 >> 172.21.15.86:0/2736171538 conn(0x7fd6240fb120 msgr2=0x7fd6240fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:21.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 -- 172.21.15.86:0/2736171538 shutdown_connections 2024-02-19T19:00:21.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 -- 172.21.15.86:0/2736171538 wait complete. 2024-02-19T19:00:21.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 Processor -- start 2024-02-19T19:00:21.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 -- start start 2024-02-19T19:00:21.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 0x7fd624195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:21.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd624100da0 0x7fd6241958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:21.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd624195ee0 con 0x7fd624100da0 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.120+0000 7fd62b323700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd624196020 con 0x7fd6240ffba0 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 0x7fd624195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 0x7fd624195380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60690/0 (socket says 172.21.15.86:60690) 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 -- 172.21.15.86:0/406868210 learned_addr learned my addr 172.21.15.86:0/406868210 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6288be700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd624100da0 0x7fd6241958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 -- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd624100da0 msgr2=0x7fd6241958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd624100da0 0x7fd6241958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 -- 172.21.15.86:0/406868210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6140095f0 con 0x7fd6240ffba0 2024-02-19T19:00:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.121+0000 7fd6290bf700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 0x7fd624195380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd62000ea80 tx=0x7fd62000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.122+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd620013070 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.123+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd62000b940 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.123+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd624068eb0 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.123+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd620003cc0 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.123+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd624069400 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.124+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd624108700 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.124+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd6200041f0 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.124+0000 7fd61a7fc700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd61006c4e0 0x7fd61006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.124+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fd62008cb50 con 0x7fd6240ffba0 2024-02-19T19:00:21.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.124+0000 7fd6288be700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd61006c4e0 0x7fd61006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:21.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.125+0000 7fd6288be700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd61006c4e0 0x7fd61006e9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd61400b6f0 tx=0x7fd614005a90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:21.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.129+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd62005b330 con 0x7fd6240ffba0 2024-02-19T19:00:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:21.330+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6241056f0 con 0x7fd61006c4e0 2024-02-19T19:00:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T19:00:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:21 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:22.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T19:00:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:21 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:23.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:22 smithi086 ceph-mon[37629]: Detected new or changed devices on smithi086 2024-02-19T19:00:23.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:22 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi086 to 7456M 2024-02-19T19:00:23.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:22 smithi086 ceph-mon[37629]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:23.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:22 smithi039 ceph-mon[28988]: Detected new or changed devices on smithi086 2024-02-19T19:00:23.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:22 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi086 to 7456M 2024-02-19T19:00:23.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:22 smithi039 ceph-mon[28988]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:24.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:23 smithi086 ceph-mon[37629]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:24.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:23 smithi039 ceph-mon[28988]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:25.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:24 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/1909715612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "772c2485-d3a7-4ee6-9f6d-4f9535db5861"}]: dispatch 2024-02-19T19:00:25.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:24 smithi039 ceph-mon[28988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "772c2485-d3a7-4ee6-9f6d-4f9535db5861"}]: dispatch 2024-02-19T19:00:25.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:24 smithi039 ceph-mon[28988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "772c2485-d3a7-4ee6-9f6d-4f9535db5861"}]': finished 2024-02-19T19:00:25.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:24 smithi039 ceph-mon[28988]: osdmap e36: 6 total, 5 up, 6 in 2024-02-19T19:00:25.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:24 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:25.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:24 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3705206359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T19:00:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:24 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/1909715612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "772c2485-d3a7-4ee6-9f6d-4f9535db5861"}]: dispatch 2024-02-19T19:00:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:24 smithi086 ceph-mon[37629]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "772c2485-d3a7-4ee6-9f6d-4f9535db5861"}]: dispatch 2024-02-19T19:00:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:24 smithi086 ceph-mon[37629]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "772c2485-d3a7-4ee6-9f6d-4f9535db5861"}]': finished 2024-02-19T19:00:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:24 smithi086 ceph-mon[37629]: osdmap e36: 6 total, 5 up, 6 in 2024-02-19T19:00:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:24 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:24 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/3705206359' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T19:00:25.794 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:25 smithi086 ceph-mon[37629]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:25.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:25 smithi039 ceph-mon[28988]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:28.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:27 smithi086 ceph-mon[37629]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:28.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:27 smithi039 ceph-mon[28988]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:30.080 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:29 smithi086 ceph-mon[37629]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:30.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:29 smithi039 ceph-mon[28988]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:30.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:31.077 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:32.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:31 smithi039 ceph-mon[28988]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:32.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:00:32.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:32.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:31 smithi086 ceph-mon[37629]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:32.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:00:32.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:32.919 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:32 smithi086 ceph-mon[37629]: Deploying daemon osd.5 on smithi086 2024-02-19T19:00:33.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:32 smithi039 ceph-mon[28988]: Deploying daemon osd.5 on smithi086 2024-02-19T19:00:33.981 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:33 smithi086 ceph-mon[37629]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:34.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:33 smithi039 ceph-mon[28988]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:35.222 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:34 smithi086 ceph-mon[37629]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:35.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:34 smithi039 ceph-mon[28988]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:37.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:37 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:37.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:37 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:37.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:37 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:37.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:37 smithi039 ceph-mon[28988]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:37.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:37 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:37.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:37 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:37.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:37 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:37.783 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:37 smithi086 ceph-mon[37629]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:39.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.943+0000 7fd61a7fc700 1 -- 172.21.15.86:0/406868210 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd6241056f0 con 0x7fd61006c4e0 2024-02-19T19:00:39.962 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd61006c4e0 msgr2=0x7fd61006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd61006c4e0 0x7fd61006e9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd61400b6f0 tx=0x7fd614005a90 comp rx=0 tx=0).stop 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 msgr2=0x7fd624195380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 0x7fd624195380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd62000ea80 tx=0x7fd62000ed90 comp rx=0 tx=0).stop 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 shutdown_connections 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd61006c4e0 0x7fd61006e9a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6240ffba0 0x7fd624195380 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:39.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 --2- 172.21.15.86:0/406868210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd624100da0 0x7fd6241958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 >> 172.21.15.86:0/406868210 conn(0x7fd6240fb120 msgr2=0x7fd624103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 shutdown_connections 2024-02-19T19:00:39.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:39.945+0000 7fd62b323700 1 -- 172.21.15.86:0/406868210 wait complete. 2024-02-19T19:00:40.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:39 smithi086 ceph-mon[37629]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:40.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:39 smithi039 ceph-mon[28988]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:40.788 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.5.service 2024-02-19T19:00:40.790 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-19T19:00:40.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-19T19:00:41.008 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:00:41.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:40 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:41.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:40 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:41.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:40 smithi086 ceph-mon[37629]: from='osd.5 [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:00:41.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:40 smithi086 ceph-mon[37629]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:00:41.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:40 smithi086 ceph-mon[37629]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:41.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:40 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:41.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:40 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:41.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:40 smithi039 ceph-mon[28988]: from='osd.5 [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:00:41.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:40 smithi039 ceph-mon[28988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:00:41.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:40 smithi039 ceph-mon[28988]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:42.177 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T19:00:42.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: osdmap e37: 6 total, 5 up, 6 in 2024-02-19T19:00:42.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:42.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: from='osd.5 [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:42.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:42.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:42.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:42.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T19:00:42.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: osdmap e37: 6 total, 5 up, 6 in 2024-02-19T19:00:42.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:42.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: from='osd.5 [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:42.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:00:42.245 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:42.246 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:42.246 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[56563]: 2024-02-19T19:00:41.907+0000 7f7b6e0f6700 -1 osd.5 0 waiting for initial osdmap 2024-02-19T19:00:42.246 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:00:41 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[56563]: 2024-02-19T19:00:41.915+0000 7f7b69eeb700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:00:42.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 -- 172.21.15.39:0/3488900681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 msgr2=0x7f91a4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3488900681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a4101270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9198009910 tx=0x7f9198009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 -- 172.21.15.39:0/3488900681 shutdown_connections 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3488900681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a4101270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3488900681 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91a40ffc10 0x7f91a4100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 -- 172.21.15.39:0/3488900681 >> 172.21.15.39:0/3488900681 conn(0x7f91a40fb190 msgr2=0x7f91a40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 -- 172.21.15.39:0/3488900681 shutdown_connections 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 -- 172.21.15.39:0/3488900681 wait complete. 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.756+0000 7f91aa9df700 1 Processor -- start 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91aa9df700 1 -- start start 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91aa9df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91a40ffc10 0x7f91a4103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91aa9df700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a41021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91aa9df700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a4102730 con 0x7f91a4100e10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91aa9df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a4102870 con 0x7f91a40ffc10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a41021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a41021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37544/0 (socket says 172.21.15.39:37544) 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 -- 172.21.15.39:0/3634476951 learned_addr learned my addr 172.21.15.39:0/3634476951 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a3fff700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91a40ffc10 0x7f91a4103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 -- 172.21.15.39:0/3634476951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91a40ffc10 msgr2=0x7f91a4103ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91a40ffc10 0x7f91a4103ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 -- 172.21.15.39:0/3634476951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980095f0 con 0x7f91a4100e10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.757+0000 7f91a37fe700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a41021f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9198006010 tx=0x7f919800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.758+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919801d070 con 0x7f91a4100e10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.758+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a4102af0 con 0x7f91a4100e10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.758+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f919800f460 con 0x7f91a4100e10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.758+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9198021640 con 0x7f91a4100e10 2024-02-19T19:00:42.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.758+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a4102fe0 con 0x7f91a4100e10 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.758+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a4108700 con 0x7f91a4100e10 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.759+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f919800fab0 con 0x7f91a4100e10 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.759+0000 7f91a17fa700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f918c06c4e0 0x7f918c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.759+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f919808c720 con 0x7f91a4100e10 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.760+0000 7f91a3fff700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f918c06c4e0 0x7f918c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.760+0000 7f91a3fff700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f918c06c4e0 0x7f918c06e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91a4100c70 tx=0x7f919400a380 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:42.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.764+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f919805ade0 con 0x7f91a4100e10 2024-02-19T19:00:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.964+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f91a419ebc0 con 0x7f91a4100e10 2024-02-19T19:00:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.965+0000 7f91a17fa700 1 -- 172.21.15.39:0/3634476951 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f919805a970 con 0x7f91a4100e10 2024-02-19T19:00:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:00:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.966+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f918c06c4e0 msgr2=0x7f918c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.966+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f918c06c4e0 0x7f918c06e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91a4100c70 tx=0x7f919400a380 comp rx=0 tx=0).stop 2024-02-19T19:00:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.966+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 msgr2=0x7f91a41021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.966+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a41021f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9198006010 tx=0x7f919800bd70 comp rx=0 tx=0).stop 2024-02-19T19:00:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 shutdown_connections 2024-02-19T19:00:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f918c06c4e0 0x7f918c06e9a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91a40ffc10 0x7f91a4103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 --2- 172.21.15.39:0/3634476951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91a4100e10 0x7f91a41021f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 >> 172.21.15.39:0/3634476951 conn(0x7f91a40fb190 msgr2=0x7f91a4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 shutdown_connections 2024-02-19T19:00:42.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:42.967+0000 7f91aa9df700 1 -- 172.21.15.39:0/3634476951 wait complete. 2024-02-19T19:00:42.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:42 smithi039 ceph-mon[28988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-02-19T19:00:42.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:42 smithi039 ceph-mon[28988]: osdmap e38: 6 total, 5 up, 6 in 2024-02-19T19:00:42.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:42 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:42.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:42 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:42.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:42 smithi039 ceph-mon[28988]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:43.223 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:42 smithi086 ceph-mon[37629]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-02-19T19:00:43.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:42 smithi086 ceph-mon[37629]: osdmap e38: 6 total, 5 up, 6 in 2024-02-19T19:00:43.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:42 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:43.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:42 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:43.224 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:42 smithi086 ceph-mon[37629]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T19:00:43.623 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1708369242,"num_in_osds":6,"osd_in_since":1708369223,"num_remapped_pgs":0} 2024-02-19T19:00:43.624 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T19:00:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:00:44.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: purged_snaps scrub starts 2024-02-19T19:00:44.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: purged_snaps scrub ok 2024-02-19T19:00:44.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:44.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: osd.5 [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] boot 2024-02-19T19:00:44.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: osdmap e39: 6 total, 6 up, 6 in 2024-02-19T19:00:44.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:44.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:43 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3634476951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: purged_snaps scrub starts 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: purged_snaps scrub ok 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: osd.5 [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] boot 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: osdmap e39: 6 total, 6 up, 6 in 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:43 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3634476951' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: osdmap e40: 6 total, 6 up, 6 in 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: Detected new or changed devices on smithi086 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi086 to 4970M 2024-02-19T19:00:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:45.152 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: osdmap e40: 6 total, 6 up, 6 in 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: Detected new or changed devices on smithi086 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi086 to 4970M 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 -- 172.21.15.39:0/1115917755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4106cd0 msgr2=0x7f92b41070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 --2- 172.21.15.39:0/1115917755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4106cd0 0x7f92b41070b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00b0a0 tx=0x7f92ac00b3b0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 -- 172.21.15.39:0/1115917755 shutdown_connections 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 --2- 172.21.15.39:0/1115917755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4068580 0x7f92b40689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 --2- 172.21.15.39:0/1115917755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4106cd0 0x7f92b41070b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 -- 172.21.15.39:0/1115917755 >> 172.21.15.39:0/1115917755 conn(0x7f92b4075670 msgr2=0x7f92b4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 -- 172.21.15.39:0/1115917755 shutdown_connections 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.629+0000 7f92bb124700 1 -- 172.21.15.39:0/1115917755 wait complete. 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92bb124700 1 Processor -- start 2024-02-19T19:00:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92bb124700 1 -- start start 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92bb124700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 0x7f92b419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92bb124700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4106cd0 0x7f92b419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92bb124700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b419d230 con 0x7f92b4068580 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92bb124700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b41966e0 con 0x7f92b4106cd0 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 0x7f92b419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 0x7f92b419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37558/0 (socket says 172.21.15.39:37558) 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 -- 172.21.15.39:0/2167853071 learned_addr learned my addr 172.21.15.39:0/2167853071 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b3fff700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4106cd0 0x7f92b419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 -- 172.21.15.39:0/2167853071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4106cd0 msgr2=0x7f92b419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4106cd0 0x7f92b419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 -- 172.21.15.39:0/2167853071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac009d00 con 0x7f92b4068580 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b3fff700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4106cd0 0x7f92b419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.630+0000 7f92b8ec0700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 0x7f92b419c660 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f92ac000f80 tx=0x7f92ac00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.631+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac00e040 con 0x7f92b4068580 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.631+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f92ac0044f0 con 0x7f92b4068580 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.631+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b4196960 con 0x7f92b4068580 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.631+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92ac01d9f0 con 0x7f92b4068580 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.631+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b4196e50 con 0x7f92b4068580 2024-02-19T19:00:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.632+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b41091f0 con 0x7f92b4068580 2024-02-19T19:00:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.632+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f92ac004660 con 0x7f92b4068580 2024-02-19T19:00:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.633+0000 7f92b1ffb700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f92a006c4e0 0x7f92a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.633+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f92ac08ce90 con 0x7f92b4068580 2024-02-19T19:00:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.633+0000 7f92b3fff700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f92a006c4e0 0x7f92a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.633+0000 7f92b3fff700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f92a006c4e0 0x7f92a006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92b4197b50 tx=0x7f929c005f50 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.637+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f92ac05b4c0 con 0x7f92b4068580 2024-02-19T19:00:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.852+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b404f2e0 con 0x7f92b4068580 2024-02-19T19:00:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.853+0000 7f92b1ffb700 1 -- 172.21.15.39:0/2167853071 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11313 (secure 0 0 0) 0x7f92ac05b050 con 0x7f92b4068580 2024-02-19T19:00:45.871 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:00:45.871 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","created":"2024-02-19T18:54:43.947412+0000","modified":"2024-02-19T19:00:43.917181+0000","last_up_change":"2024-02-19T19:00:42.906219+0000","last_in_change":"2024-02-19T19:00:23.810529+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:59:26.986411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91f217d9-d1a8-4658-b693-873f74d42408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6803","nonce":2663416145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6805","nonce":2663416145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6809","nonce":2663416145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6807","nonce":2663416145}]},"public_addr":"172.21.15.39:6803/2663416145","cluster_addr":"172.21.15.39:6805/2663416145","heartbeat_back_addr":"172.21.15.39:6809/2663416145","heartbeat_front_addr":"172.21.15.39:6807/2663416145","state":["exists","up"]},{"osd":1,"uuid":"4b6143b3-8bce-4be7-a199-57711f652a9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6811","nonce":3708594483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6813","nonce":3708594483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6817","nonce":3708594483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6815","nonce":3708594483}]},"public_addr":"172.21.15.39:6811/3708594483","cluster_addr":"172.21.15.39:6813/3708594483","heartbeat_back_addr":"172.21.15.39:6817/3708594483","heartbeat_front_addr":"172.21.15.39:6815/3708594483","state":["exists","up"]},{"osd":2,"uuid":"bcb75e1b-3f22-4e89-9bdd-ce8e833a3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6819","nonce":3928116015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6821","nonce":3928116015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6825","nonce":3928116015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6823","nonce":3928116015}]},"public_addr":"172.21.15.39:6819/3928116015","cluster_addr":"172.21.15.39:6821/3928116015","heartbeat_back_addr":"172.21.15.39:6825/3928116015","heartbeat_front_addr":"172.21.15.39:6823/3928116015","state":["exists","up"]},{"osd":3,"uuid":"8c3353e0-09f3-4582-af41-33ed9e1581c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6801","nonce":1886206958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6803","nonce":1886206958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6807","nonce":1886206958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6805","nonce":1886206958}]},"public_addr":"172.21.15.86:6801/1886206958","cluster_addr":"172.21.15.86:6803/1886206958","heartbeat_back_addr":"172.21.15.86:6807/1886206958","heartbeat_front_addr":"172.21.15.86:6805/1886206958","state":["exists","up"]},{"osd":4,"uuid":"62e52e70-7bbd-4395-98b1-2bc8eabc2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6809","nonce":2218646412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6811","nonce":2218646412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6815","nonce":2218646412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6813","nonce":2218646412}]},"public_addr":"172.21.15.86:6809/2218646412","cluster_addr":"172.21.15.86:6811/2218646412","heartbeat_back_addr":"172.21.15.86:6815/2218646412","heartbeat_front_addr":"172.21.15.86:6813/2218646412","state":["exists","up"]},{"osd":5,"uuid":"772c2485-d3a7-4ee6-9f6d-4f9535db5861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6817","nonce":1712053993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6819","nonce":1712053993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6823","nonce":1712053993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6821","nonce":1712053993}]},"public_addr":"172.21.15.86:6817/1712053993","cluster_addr":"172.21.15.86:6819/1712053993","heartbeat_back_addr":"172.21.15.86:6823/1712053993","heartbeat_front_addr":"172.21.15.86:6821/1712053993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:24.714559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:54.443434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:24.712567+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:49.208459+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:15.215844+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:41.317444+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3933707157":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2619661656":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2313120383":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1749005598":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6800/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1261064923":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1614648654":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/3613879910":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6801/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2871115404":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1444159856":"2024-02-20T18:55:10.759262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f92a006c4e0 msgr2=0x7f92a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f92a006c4e0 0x7f92a006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92b4197b50 tx=0x7f929c005f50 comp rx=0 tx=0).stop 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 msgr2=0x7f92b419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 0x7f92b419c660 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f92ac000f80 tx=0x7f92ac00cdb0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 shutdown_connections 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f92a006c4e0 0x7f92a006e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b4068580 0x7f92b419c660 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 --2- 172.21.15.39:0/2167853071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92b4106cd0 0x7f92b419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 >> 172.21.15.39:0/2167853071 conn(0x7f92b4075670 msgr2=0x7f92b40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 shutdown_connections 2024-02-19T19:00:45.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:45.855+0000 7f92bb124700 1 -- 172.21.15.39:0/2167853071 wait complete. 2024-02-19T19:00:46.696 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:59:26.986411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T19:00:46.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-19T19:00:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:00:46.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:46 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:46.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:46 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2167853071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:00:47.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:46 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:47.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:46 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2167853071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:00:47.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:47 smithi039 ceph-mon[28988]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:48.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:47 smithi086 ceph-mon[37629]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.587+0000 7f89c4926700 1 -- 172.21.15.39:0/461778197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 msgr2=0x7f89c0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.587+0000 7f89c4926700 1 --2- 172.21.15.39:0/461778197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0101280 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009910 tx=0x7f89b0009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 -- 172.21.15.39:0/461778197 shutdown_connections 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 --2- 172.21.15.39:0/461778197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89c0101850 0x7f89c01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 --2- 172.21.15.39:0/461778197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0101280 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 -- 172.21.15.39:0/461778197 >> 172.21.15.39:0/461778197 conn(0x7f89c00fc750 msgr2=0x7f89c00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 -- 172.21.15.39:0/461778197 shutdown_connections 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 -- 172.21.15.39:0/461778197 wait complete. 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 Processor -- start 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.588+0000 7f89c4926700 1 -- start start 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89c4926700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89c4926700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89c0101850 0x7f89c00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89c4926700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0073b10 con 0x7f89c0100ea0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89c4926700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c0073c50 con 0x7f89c0101850 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46432/0 (socket says 172.21.15.39:46432) 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 -- 172.21.15.39:0/3899005431 learned_addr learned my addr 172.21.15.39:0/3899005431 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 -- 172.21.15.39:0/3899005431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89c0101850 msgr2=0x7f89c00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89c0101850 0x7f89c00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 -- 172.21.15.39:0/3899005431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b00095f0 con 0x7f89c0100ea0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89be59c700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0073090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006010 tx=0x7f89b000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89b001d070 con 0x7f89c0100ea0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f89b000f460 con 0x7f89c0100ea0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c0073ed0 con 0x7f89c0100ea0 2024-02-19T19:00:48.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f89b0021640 con 0x7f89c0100ea0 2024-02-19T19:00:48.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.589+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c01a2e20 con 0x7f89c0100ea0 2024-02-19T19:00:48.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.590+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c0109240 con 0x7f89c0100ea0 2024-02-19T19:00:48.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.591+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f89b000faf0 con 0x7f89c0100ea0 2024-02-19T19:00:48.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.591+0000 7f89af7fe700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f89a806c4e0 0x7f89a806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:48.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.591+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f89b008c7e0 con 0x7f89c0100ea0 2024-02-19T19:00:48.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.591+0000 7f89bdd9b700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f89a806c4e0 0x7f89a806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:48.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.592+0000 7f89bdd9b700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f89a806c4e0 0x7f89a806e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89c0074880 tx=0x7f89b4005c80 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:48.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.596+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f89b0057300 con 0x7f89c0100ea0 2024-02-19T19:00:48.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.792+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89c0108c20 con 0x7f89c0100ea0 2024-02-19T19:00:48.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.793+0000 7f89af7fe700 1 -- 172.21.15.39:0/3899005431 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f89b005a920 con 0x7f89c0100ea0 2024-02-19T19:00:48.811 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f89a806c4e0 msgr2=0x7f89a806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f89a806c4e0 0x7f89a806e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89c0074880 tx=0x7f89b4005c80 comp rx=0 tx=0).stop 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 msgr2=0x7f89c0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0073090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006010 tx=0x7f89b000be20 comp rx=0 tx=0).stop 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 shutdown_connections 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f89a806c4e0 0x7f89a806e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89c0100ea0 0x7f89c0073090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 --2- 172.21.15.39:0/3899005431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89c0101850 0x7f89c00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 >> 172.21.15.39:0/3899005431 conn(0x7f89c00fc750 msgr2=0x7f89c00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:48.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 shutdown_connections 2024-02-19T19:00:48.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:48.795+0000 7f89c4926700 1 -- 172.21.15.39:0/3899005431 wait complete. 2024-02-19T19:00:49.696 INFO:tasks.cephadm:Setting up client nodes... 2024-02-19T19:00:49.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T19:00:49.909 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:00:49.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:49 smithi039 ceph-mon[28988]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:49.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:49 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3899005431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:00:50.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:49 smithi086 ceph-mon[37629]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:50.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:49 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3899005431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:00:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 -- 172.21.15.39:0/4016219273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 msgr2=0x7f15801015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 --2- 172.21.15.39:0/4016219273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f15801015d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1570009910 tx=0x7f1570009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 -- 172.21.15.39:0/4016219273 shutdown_connections 2024-02-19T19:00:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 --2- 172.21.15.39:0/4016219273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15800ffe10 0x7f1580100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 --2- 172.21.15.39:0/4016219273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f15801015d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 -- 172.21.15.39:0/4016219273 >> 172.21.15.39:0/4016219273 conn(0x7f15800fa8c0 msgr2=0x7f15800fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 -- 172.21.15.39:0/4016219273 shutdown_connections 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 -- 172.21.15.39:0/4016219273 wait complete. 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 Processor -- start 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.586+0000 7f1586509700 1 -- start start 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f1586509700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15800ffe10 0x7f15800ff860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f1586509700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f158019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f1586509700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158019b860 con 0x7f15801011b0 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f1586509700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158019b9a0 con 0x7f15800ffe10 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f158019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f158019b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46456/0 (socket says 172.21.15.39:46456) 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 -- 172.21.15.39:0/3728513797 learned_addr learned my addr 172.21.15.39:0/3728513797 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:00:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157ffff700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15800ffe10 0x7f15800ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 -- 172.21.15.39:0/3728513797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15800ffe10 msgr2=0x7f15800ff860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15800ffe10 0x7f15800ff860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 -- 172.21.15.39:0/3728513797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700095f0 con 0x7f15801011b0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157f7fe700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f158019b290 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f157400b4b0 tx=0x7f157400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1574011800 con 0x7f15801011b0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1574011e10 con 0x7f15801011b0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158019bc80 con 0x7f15801011b0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.587+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f157400f2e0 con 0x7f15801011b0 2024-02-19T19:00:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.588+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158019c1d0 con 0x7f15801011b0 2024-02-19T19:00:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.588+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580108790 con 0x7f15801011b0 2024-02-19T19:00:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.589+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1574010380 con 0x7f15801011b0 2024-02-19T19:00:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.589+0000 7f157d7fa700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f156806c4e0 0x7f156806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.589+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f157408b060 con 0x7f15801011b0 2024-02-19T19:00:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.589+0000 7f157ffff700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f156806c4e0 0x7f156806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.590+0000 7f157ffff700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f156806c4e0 0x7f156806e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1570006010 tx=0x7f157000b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:51.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.594+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1574059610 con 0x7f15801011b0 2024-02-19T19:00:51.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:51 smithi039 ceph-mon[28988]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:51.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.874+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f158019c840 con 0x7f15801011b0 2024-02-19T19:00:51.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.878+0000 7f157d7fa700 1 -- 172.21.15.39:0/3728513797 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f15740591a0 con 0x7f15801011b0 2024-02-19T19:00:51.897 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-02-19T19:00:51.898 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBjpdNl7rk4NBAAYduMq1FPrLiwL2FtmMR3CA== 2024-02-19T19:00:51.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f156806c4e0 msgr2=0x7f156806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f156806c4e0 0x7f156806e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1570006010 tx=0x7f157000b540 comp rx=0 tx=0).stop 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 msgr2=0x7f158019b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f158019b290 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f157400b4b0 tx=0x7f157400b7c0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 shutdown_connections 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f156806c4e0 0x7f156806e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15800ffe10 0x7f15800ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 --2- 172.21.15.39:0/3728513797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15801011b0 0x7f158019b290 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 >> 172.21.15.39:0/3728513797 conn(0x7f15800fa8c0 msgr2=0x7f1580104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 shutdown_connections 2024-02-19T19:00:51.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:51.880+0000 7f1586509700 1 -- 172.21.15.39:0/3728513797 wait complete. 2024-02-19T19:00:52.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:51 smithi086 ceph-mon[37629]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:52.602 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T19:00:52.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-19T19:00:52.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-19T19:00:52.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T19:00:52.865 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi086/config 2024-02-19T19:00:52.986 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:52 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3728513797' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T19:00:52.986 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:52 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3728513797' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T19:00:52.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:52 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3728513797' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T19:00:52.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:52 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3728513797' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T19:00:53.977 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:53 smithi086 ceph-mon[37629]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:53.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:53 smithi039 ceph-mon[28988]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:54.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 -- 172.21.15.86:0/2887109243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 msgr2=0x7f9f0c0ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:54.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/2887109243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c0ff380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009910 tx=0x7f9efc009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:54.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 -- 172.21.15.86:0/2887109243 shutdown_connections 2024-02-19T19:00:54.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/2887109243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0c101f40 0x7f9f0c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/2887109243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c0ff380 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 -- 172.21.15.86:0/2887109243 >> 172.21.15.86:0/2887109243 conn(0x7f9f0c0fa8c0 msgr2=0x7f9f0c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 -- 172.21.15.86:0/2887109243 shutdown_connections 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.479+0000 7f9f10ddb700 1 -- 172.21.15.86:0/2887109243 wait complete. 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f10ddb700 1 Processor -- start 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f10ddb700 1 -- start start 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f10ddb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f10ddb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0c101f40 0x7f9f0c0ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f10ddb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c100430 con 0x7f9f0c101f40 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f10ddb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c100570 con 0x7f9f0c0fef60 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:41968/0 (socket says 172.21.15.86:41968) 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 -- 172.21.15.86:0/3606888462 learned_addr learned my addr 172.21.15.86:0/3606888462 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-02-19T19:00:54.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f09d9b700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0c101f40 0x7f9f0c0ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 -- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0c101f40 msgr2=0x7f9f0c0ffef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0c101f40 0x7f9f0c0ffef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 -- 172.21.15.86:0/3606888462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0095f0 con 0x7f9f0c0fef60 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9f0a59c700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c1018a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00bda0 tx=0x7f9efc00bdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.480+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9efc01d070 con 0x7f9f0c0fef60 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.481+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9efc00f460 con 0x7f9f0c0fef60 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.481+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c1007f0 con 0x7f9f0c0fef60 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.481+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9efc021640 con 0x7f9f0c0fef60 2024-02-19T19:00:54.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.481+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c100ce0 con 0x7f9f0c0fef60 2024-02-19T19:00:54.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.482+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c109800 con 0x7f9f0c0fef60 2024-02-19T19:00:54.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.482+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f9efc003a60 con 0x7f9f0c0fef60 2024-02-19T19:00:54.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.482+0000 7f9efb7fe700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f9ef406c4e0 0x7f9ef406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:54.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.482+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9efc08c690 con 0x7f9f0c0fef60 2024-02-19T19:00:54.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.483+0000 7f9f09d9b700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f9ef406c4e0 0x7f9ef406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:54.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.483+0000 7f9f09d9b700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f9ef406c4e0 0x7f9ef406e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f00007be0 tx=0x7f9f000073d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:54.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.487+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9efc05acc0 con 0x7f9f0c0fef60 2024-02-19T19:00:54.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.754+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9f0c1099f0 con 0x7f9f0c0fef60 2024-02-19T19:00:54.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.760+0000 7f9efb7fe700 1 -- 172.21.15.86:0/3606888462 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f9efc05a850 con 0x7f9f0c0fef60 2024-02-19T19:00:54.778 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-02-19T19:00:54.778 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBmpdNlI+gWLRAA0DSv1yr5d7wKXo7zG8UwqA== 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f9ef406c4e0 msgr2=0x7f9ef406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f9ef406c4e0 0x7f9ef406e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f00007be0 tx=0x7f9f000073d0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 msgr2=0x7f9f0c1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c1018a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00bda0 tx=0x7f9efc00bdd0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 shutdown_connections 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f9ef406c4e0 0x7f9ef406e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0fef60 0x7f9f0c1018a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 --2- 172.21.15.86:0/3606888462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0c101f40 0x7f9f0c0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 >> 172.21.15.86:0/3606888462 conn(0x7f9f0c0fa8c0 msgr2=0x7f9f0c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:54.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 shutdown_connections 2024-02-19T19:00:54.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-02-19T19:00:54.762+0000 7f9f10ddb700 1 -- 172.21.15.86:0/3606888462 wait complete. 2024-02-19T19:00:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:54 smithi086 ceph-mon[37629]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:54 smithi086 ceph-mon[37629]: from='client.? 172.21.15.86:0/3606888462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T19:00:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:54 smithi086 ceph-mon[37629]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T19:00:55.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:54 smithi086 ceph-mon[37629]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T19:00:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:54 smithi039 ceph-mon[28988]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:54 smithi039 ceph-mon[28988]: from='client.? 172.21.15.86:0/3606888462' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T19:00:55.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:54 smithi039 ceph-mon[28988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T19:00:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:54 smithi039 ceph-mon[28988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T19:00:55.522 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T19:00:55.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-19T19:00:55.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-19T19:00:55.565 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-19T19:00:55.565 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-19T19:00:55.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph mgr dump --format=json 2024-02-19T19:00:55.784 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 -- 172.21.15.39:0/2348175662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 msgr2=0x7f4d3c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2348175662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c1047a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009910 tx=0x7f4d30009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 -- 172.21.15.39:0/2348175662 shutdown_connections 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2348175662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c1047a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2348175662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d3c0ffda0 0x7f4d3c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 -- 172.21.15.39:0/2348175662 >> 172.21.15.39:0/2348175662 conn(0x7f4d3c0fb650 msgr2=0x7f4d3c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 -- 172.21.15.39:0/2348175662 shutdown_connections 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 -- 172.21.15.39:0/2348175662 wait complete. 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 Processor -- start 2024-02-19T19:00:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.498+0000 7f4d445bf700 1 -- start start 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d445bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d3c0ffda0 0x7f4d3c19b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d445bf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c19bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d445bf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c19c1f0 con 0x7f4d3c100750 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d445bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c1956a0 con 0x7f4d3c0ffda0 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c19bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c19bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35246/0 (socket says 172.21.15.39:35246) 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 -- 172.21.15.39:0/2983347582 learned_addr learned my addr 172.21.15.39:0/2983347582 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d4235b700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d3c0ffda0 0x7f4d3c19b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 -- 172.21.15.39:0/2983347582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d3c0ffda0 msgr2=0x7f4d3c19b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d3c0ffda0 0x7f4d3c19b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 -- 172.21.15.39:0/2983347582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300095f0 con 0x7f4d3c100750 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d41b5a700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c19bb60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009fd0 tx=0x7f4d3000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d3001d070 con 0x7f4d3c100750 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d30005640 con 0x7f4d3c100750 2024-02-19T19:00:57.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3c195920 con 0x7f4d3c100750 2024-02-19T19:00:57.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d30022470 con 0x7f4d3c100750 2024-02-19T19:00:57.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.499+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3c195e10 con 0x7f4d3c100750 2024-02-19T19:00:57.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.500+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3c1081a0 con 0x7f4d3c100750 2024-02-19T19:00:57.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.501+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4d30003680 con 0x7f4d3c100750 2024-02-19T19:00:57.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.501+0000 7f4d3b7fe700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4d2406c4e0 0x7f4d2406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.501+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4d3008c900 con 0x7f4d3c100750 2024-02-19T19:00:57.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.501+0000 7f4d4235b700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4d2406c4e0 0x7f4d2406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.502+0000 7f4d4235b700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4d2406c4e0 0x7f4d2406e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400f540 tx=0x7f4d3400b040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.506+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4d3005aeb0 con 0x7f4d3c100750 2024-02-19T19:00:57.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.763+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4d3c04f2e0 con 0x7f4d3c100750 2024-02-19T19:00:57.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.766+0000 7f4d3b7fe700 1 -- 172.21.15.39:0/2983347582 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173038 (secure 0 0 0) 0x7f4d3005aa40 con 0x7f4d3c100750 2024-02-19T19:00:57.789 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:00:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4d2406c4e0 msgr2=0x7f4d2406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4d2406c4e0 0x7f4d2406e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400f540 tx=0x7f4d3400b040 comp rx=0 tx=0).stop 2024-02-19T19:00:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 msgr2=0x7f4d3c19bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c19bb60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009fd0 tx=0x7f4d3000fa00 comp rx=0 tx=0).stop 2024-02-19T19:00:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 shutdown_connections 2024-02-19T19:00:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f4d2406c4e0 0x7f4d2406e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d3c0ffda0 0x7f4d3c19b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 --2- 172.21.15.39:0/2983347582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d3c100750 0x7f4d3c19bb60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.769+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 >> 172.21.15.39:0/2983347582 conn(0x7f4d3c0fb650 msgr2=0x7f4d3c0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.770+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 shutdown_connections 2024-02-19T19:00:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:00:57.770+0000 7f4d445bf700 1 -- 172.21.15.39:0/2983347582 wait complete. 2024-02-19T19:00:58.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:57 smithi086 ceph-mon[37629]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:58.092 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:57 smithi039 ceph-mon[28988]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:58.532 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":19,"active_gid":14219,"active_name":"smithi039.ydmpuh","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2307886882},{"type":"v1","addr":"172.21.15.39:6801","nonce":2307886882}]},"active_addr":"172.21.15.39:6801/2307886882","active_change":"2024-02-19T18:57:00.645575+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14246,"name":"smithi086.huubbu","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.39:8443/","prometheus":"http://172.21.15.39:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3009513678}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":883565860}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3788984480}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":2615309126}]}]} 2024-02-19T19:00:58.536 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-19T19:00:58.536 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-19T19:00:58.537 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T19:00:58.746 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:00:59.021 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:58 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2983347582' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T19:00:59.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:58 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2983347582' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T19:00:59.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:00:59 smithi039 ceph-mon[28988]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:00.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:00:59 smithi086 ceph-mon[37629]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.342+0000 7fbf6114c700 1 -- 172.21.15.39:0/1680677501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c100750 msgr2=0x7fbf5c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.342+0000 7fbf6114c700 1 --2- 172.21.15.39:0/1680677501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c100750 0x7fbf5c1047a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009910 tx=0x7fbf50009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.342+0000 7fbf6114c700 1 -- 172.21.15.39:0/1680677501 shutdown_connections 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.342+0000 7fbf6114c700 1 --2- 172.21.15.39:0/1680677501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c100750 0x7fbf5c1047a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.342+0000 7fbf6114c700 1 --2- 172.21.15.39:0/1680677501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.342+0000 7fbf6114c700 1 -- 172.21.15.39:0/1680677501 >> 172.21.15.39:0/1680677501 conn(0x7fbf5c0fb650 msgr2=0x7fbf5c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.343+0000 7fbf6114c700 1 -- 172.21.15.39:0/1680677501 shutdown_connections 2024-02-19T19:01:00.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.343+0000 7fbf6114c700 1 -- 172.21.15.39:0/1680677501 wait complete. 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.343+0000 7fbf6114c700 1 Processor -- start 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.343+0000 7fbf6114c700 1 -- start start 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf6114c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf6114c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c100750 0x7fbf5c19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf6114c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c19d240 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf6114c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c1966f0 con 0x7fbf5c100750 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35266/0 (socket says 172.21.15.39:35266) 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 -- 172.21.15.39:0/772688291 learned_addr learned my addr 172.21.15.39:0/772688291 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:00.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5a59c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c100750 0x7fbf5c19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 -- 172.21.15.39:0/772688291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c100750 msgr2=0x7fbf5c19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c100750 0x7fbf5c19cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 -- 172.21.15.39:0/772688291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500095f0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5a59c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c100750 0x7fbf5c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf5ad9d700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c19c670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00d8d0 tx=0x7fbf4c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4c009940 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbf4c010460 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.344+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c1969d0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.345+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c196f20 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.345+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4c00f5f0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.345+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c1091b0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.346+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fbf4c0105d0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.346+0000 7fbf43fff700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fbf4406c4e0 0x7fbf4406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:00.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.347+0000 7fbf5a59c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fbf4406c4e0 0x7fbf4406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:00.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.347+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbf4c08b4f0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.347+0000 7fbf5a59c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fbf4406c4e0 0x7fbf4406e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf50006010 tx=0x7fbf50005960 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:00.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.351+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbf4c059aa0 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.548+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf5c197d00 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.548+0000 7fbf43fff700 1 -- 172.21.15.39:0/772688291 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11313 (secure 0 0 0) 0x7fbf5c197d00 con 0x7fbf5c0ffda0 2024-02-19T19:01:00.568 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:01:00.568 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","created":"2024-02-19T18:54:43.947412+0000","modified":"2024-02-19T19:00:43.917181+0000","last_up_change":"2024-02-19T19:00:42.906219+0000","last_in_change":"2024-02-19T19:00:23.810529+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:59:26.986411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91f217d9-d1a8-4658-b693-873f74d42408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6803","nonce":2663416145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6805","nonce":2663416145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6809","nonce":2663416145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6807","nonce":2663416145}]},"public_addr":"172.21.15.39:6803/2663416145","cluster_addr":"172.21.15.39:6805/2663416145","heartbeat_back_addr":"172.21.15.39:6809/2663416145","heartbeat_front_addr":"172.21.15.39:6807/2663416145","state":["exists","up"]},{"osd":1,"uuid":"4b6143b3-8bce-4be7-a199-57711f652a9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6811","nonce":3708594483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6813","nonce":3708594483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6817","nonce":3708594483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6815","nonce":3708594483}]},"public_addr":"172.21.15.39:6811/3708594483","cluster_addr":"172.21.15.39:6813/3708594483","heartbeat_back_addr":"172.21.15.39:6817/3708594483","heartbeat_front_addr":"172.21.15.39:6815/3708594483","state":["exists","up"]},{"osd":2,"uuid":"bcb75e1b-3f22-4e89-9bdd-ce8e833a3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6819","nonce":3928116015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6821","nonce":3928116015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6825","nonce":3928116015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6823","nonce":3928116015}]},"public_addr":"172.21.15.39:6819/3928116015","cluster_addr":"172.21.15.39:6821/3928116015","heartbeat_back_addr":"172.21.15.39:6825/3928116015","heartbeat_front_addr":"172.21.15.39:6823/3928116015","state":["exists","up"]},{"osd":3,"uuid":"8c3353e0-09f3-4582-af41-33ed9e1581c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6801","nonce":1886206958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6803","nonce":1886206958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6807","nonce":1886206958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6805","nonce":1886206958}]},"public_addr":"172.21.15.86:6801/1886206958","cluster_addr":"172.21.15.86:6803/1886206958","heartbeat_back_addr":"172.21.15.86:6807/1886206958","heartbeat_front_addr":"172.21.15.86:6805/1886206958","state":["exists","up"]},{"osd":4,"uuid":"62e52e70-7bbd-4395-98b1-2bc8eabc2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6809","nonce":2218646412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6811","nonce":2218646412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6815","nonce":2218646412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6813","nonce":2218646412}]},"public_addr":"172.21.15.86:6809/2218646412","cluster_addr":"172.21.15.86:6811/2218646412","heartbeat_back_addr":"172.21.15.86:6815/2218646412","heartbeat_front_addr":"172.21.15.86:6813/2218646412","state":["exists","up"]},{"osd":5,"uuid":"772c2485-d3a7-4ee6-9f6d-4f9535db5861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6817","nonce":1712053993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6819","nonce":1712053993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6823","nonce":1712053993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6821","nonce":1712053993}]},"public_addr":"172.21.15.86:6817/1712053993","cluster_addr":"172.21.15.86:6819/1712053993","heartbeat_back_addr":"172.21.15.86:6823/1712053993","heartbeat_front_addr":"172.21.15.86:6821/1712053993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:24.714559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:54.443434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:24.712567+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:49.208459+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:15.215844+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:41.317444+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3933707157":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2619661656":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2313120383":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1749005598":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6800/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1261064923":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1614648654":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/3613879910":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6801/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2871115404":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1444159856":"2024-02-20T18:55:10.759262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:01:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fbf4406c4e0 msgr2=0x7fbf4406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fbf4406c4e0 0x7fbf4406e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf50006010 tx=0x7fbf50005960 comp rx=0 tx=0).stop 2024-02-19T19:01:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 msgr2=0x7fbf5c19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c19c670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00d8d0 tx=0x7fbf4c00dc90 comp rx=0 tx=0).stop 2024-02-19T19:01:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 shutdown_connections 2024-02-19T19:01:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fbf4406c4e0 0x7fbf4406e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:00.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf5c0ffda0 0x7fbf5c19c670 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:00.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 --2- 172.21.15.39:0/772688291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf5c100750 0x7fbf5c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:00.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 >> 172.21.15.39:0/772688291 conn(0x7fbf5c0fb650 msgr2=0x7fbf5c069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:00.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 shutdown_connections 2024-02-19T19:01:00.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:00.551+0000 7fbf6114c700 1 -- 172.21.15.39:0/772688291 wait complete. 2024-02-19T19:01:00.929 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:00 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/772688291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:01:00.929 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:01.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:00 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/772688291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:01:01.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:01.321 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-19T19:01:01.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T19:01:01.532 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:01.969 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:01 smithi039 ceph-mon[28988]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:02.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:01 smithi086 ceph-mon[37629]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 -- 172.21.15.39:0/2221908534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 msgr2=0x7f8894101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 --2- 172.21.15.39:0/2221908534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894101270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8884009910 tx=0x7f8884009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 -- 172.21.15.39:0/2221908534 shutdown_connections 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 --2- 172.21.15.39:0/2221908534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8894101840 0x7f8894105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 --2- 172.21.15.39:0/2221908534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894101270 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 -- 172.21.15.39:0/2221908534 >> 172.21.15.39:0/2221908534 conn(0x7f88940fc740 msgr2=0x7f88940feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 -- 172.21.15.39:0/2221908534 shutdown_connections 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 -- 172.21.15.39:0/2221908534 wait complete. 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.192+0000 7f889a927700 1 Processor -- start 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f889a927700 1 -- start start 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f889a927700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f889a927700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8894101840 0x7f8894196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f889a927700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8894197550 con 0x7f8894100e90 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f889a927700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889419b360 con 0x7f8894101840 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35284/0 (socket says 172.21.15.39:35284) 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 -- 172.21.15.39:0/830827307 learned_addr learned my addr 172.21.15.39:0/830827307 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f88937fe700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8894101840 0x7f8894196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 -- 172.21.15.39:0/830827307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8894101840 msgr2=0x7f8894196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8894101840 0x7f8894196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 -- 172.21.15.39:0/830827307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88840095f0 con 0x7f8894100e90 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.193+0000 7f8893fff700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894196980 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8884000c00 tx=0x7f888400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.194+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f888401d070 con 0x7f8894100e90 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.194+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f888400f460 con 0x7f8894100e90 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.194+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889419b5e0 con 0x7f8894100e90 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.194+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8884017630 con 0x7f8894100e90 2024-02-19T19:01:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.194+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889419bad0 con 0x7f8894100e90 2024-02-19T19:01:03.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.195+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8894109230 con 0x7f8894100e90 2024-02-19T19:01:03.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.195+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f888400f5d0 con 0x7f8894100e90 2024-02-19T19:01:03.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.196+0000 7f88917fa700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f887c06c4e0 0x7f887c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:03.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.196+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f888408cb20 con 0x7f8894100e90 2024-02-19T19:01:03.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.196+0000 7f88937fe700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f887c06c4e0 0x7f887c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:03.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.196+0000 7f88937fe700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f887c06c4e0 0x7f887c06e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:03.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.200+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f888405b0d0 con 0x7f8894100e90 2024-02-19T19:01:03.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.398+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8894197ce0 con 0x7f8894100e90 2024-02-19T19:01:03.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.398+0000 7f88917fa700 1 -- 172.21.15.39:0/830827307 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11313 (secure 0 0 0) 0x7f888405ac60 con 0x7f8894100e90 2024-02-19T19:01:03.418 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:01:03.418 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","created":"2024-02-19T18:54:43.947412+0000","modified":"2024-02-19T19:00:43.917181+0000","last_up_change":"2024-02-19T19:00:42.906219+0000","last_in_change":"2024-02-19T19:00:23.810529+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:59:26.986411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91f217d9-d1a8-4658-b693-873f74d42408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6803","nonce":2663416145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6805","nonce":2663416145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6809","nonce":2663416145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6807","nonce":2663416145}]},"public_addr":"172.21.15.39:6803/2663416145","cluster_addr":"172.21.15.39:6805/2663416145","heartbeat_back_addr":"172.21.15.39:6809/2663416145","heartbeat_front_addr":"172.21.15.39:6807/2663416145","state":["exists","up"]},{"osd":1,"uuid":"4b6143b3-8bce-4be7-a199-57711f652a9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6811","nonce":3708594483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6813","nonce":3708594483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6817","nonce":3708594483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6815","nonce":3708594483}]},"public_addr":"172.21.15.39:6811/3708594483","cluster_addr":"172.21.15.39:6813/3708594483","heartbeat_back_addr":"172.21.15.39:6817/3708594483","heartbeat_front_addr":"172.21.15.39:6815/3708594483","state":["exists","up"]},{"osd":2,"uuid":"bcb75e1b-3f22-4e89-9bdd-ce8e833a3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6819","nonce":3928116015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6821","nonce":3928116015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6825","nonce":3928116015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6823","nonce":3928116015}]},"public_addr":"172.21.15.39:6819/3928116015","cluster_addr":"172.21.15.39:6821/3928116015","heartbeat_back_addr":"172.21.15.39:6825/3928116015","heartbeat_front_addr":"172.21.15.39:6823/3928116015","state":["exists","up"]},{"osd":3,"uuid":"8c3353e0-09f3-4582-af41-33ed9e1581c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6801","nonce":1886206958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6803","nonce":1886206958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6807","nonce":1886206958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6805","nonce":1886206958}]},"public_addr":"172.21.15.86:6801/1886206958","cluster_addr":"172.21.15.86:6803/1886206958","heartbeat_back_addr":"172.21.15.86:6807/1886206958","heartbeat_front_addr":"172.21.15.86:6805/1886206958","state":["exists","up"]},{"osd":4,"uuid":"62e52e70-7bbd-4395-98b1-2bc8eabc2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6809","nonce":2218646412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6811","nonce":2218646412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6815","nonce":2218646412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6813","nonce":2218646412}]},"public_addr":"172.21.15.86:6809/2218646412","cluster_addr":"172.21.15.86:6811/2218646412","heartbeat_back_addr":"172.21.15.86:6815/2218646412","heartbeat_front_addr":"172.21.15.86:6813/2218646412","state":["exists","up"]},{"osd":5,"uuid":"772c2485-d3a7-4ee6-9f6d-4f9535db5861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6817","nonce":1712053993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6819","nonce":1712053993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6823","nonce":1712053993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6821","nonce":1712053993}]},"public_addr":"172.21.15.86:6817/1712053993","cluster_addr":"172.21.15.86:6819/1712053993","heartbeat_back_addr":"172.21.15.86:6823/1712053993","heartbeat_front_addr":"172.21.15.86:6821/1712053993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:24.714559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:54.443434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:24.712567+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:49.208459+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:15.215844+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:41.317444+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3933707157":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2619661656":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2313120383":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1749005598":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6800/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1261064923":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1614648654":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/3613879910":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6801/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2871115404":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1444159856":"2024-02-20T18:55:10.759262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.400+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f887c06c4e0 msgr2=0x7f887c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.400+0000 7f889a927700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f887c06c4e0 0x7f887c06e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8888006fd0 tx=0x7f8888008040 comp rx=0 tx=0).stop 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.400+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 msgr2=0x7f8894196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.400+0000 7f889a927700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894196980 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8884000c00 tx=0x7f888400be20 comp rx=0 tx=0).stop 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 shutdown_connections 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f887c06c4e0 0x7f887c06e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8894100e90 0x7f8894196980 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 --2- 172.21.15.39:0/830827307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8894101840 0x7f8894196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 >> 172.21.15.39:0/830827307 conn(0x7f88940fc740 msgr2=0x7f88940fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 shutdown_connections 2024-02-19T19:01:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:03.401+0000 7f889a927700 1 -- 172.21.15.39:0/830827307 wait complete. 2024-02-19T19:01:03.971 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:03 smithi039 ceph-mon[28988]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:03.971 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:03 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/830827307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:01:03.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-19T19:01:03.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-19T19:01:03.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-19T19:01:03.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-19T19:01:03.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-19T19:01:03.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-19T19:01:04.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:03 smithi086 ceph-mon[37629]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:04.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:03 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/830827307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:01:04.315 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:04.315 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:05.191 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:05.192 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:05.192 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:05.193 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:05.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:04 smithi039 ceph-mon[28988]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:05.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:04 smithi086 ceph-mon[37629]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:07.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:07 smithi039 ceph-mon[28988]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:08.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:07 smithi086 ceph-mon[37629]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:09.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.724+0000 7fa13d1e3700 1 -- 172.21.15.39:0/1979326767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 msgr2=0x7fa138100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.724+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/1979326767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138100080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009910 tx=0x7fa12c009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 -- 172.21.15.39:0/1979326767 shutdown_connections 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/1979326767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138100080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/1979326767 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138105bd0 0x7fa138105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 -- 172.21.15.39:0/1979326767 >> 172.21.15.39:0/1979326767 conn(0x7fa138075670 msgr2=0x7fa138075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 -- 172.21.15.39:0/1979326767 shutdown_connections 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 -- 172.21.15.39:0/1979326767 wait complete. 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 Processor -- start 2024-02-19T19:01:09.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.725+0000 7fa13d1e3700 1 -- start start 2024-02-19T19:01:09.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa13d1e3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:09.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa13d1e3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138105bd0 0x7fa138101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:09.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa13d1e3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1381022c0 con 0x7fa1380ffc20 2024-02-19T19:01:09.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa13d1e3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa138102400 con 0x7fa138105bd0 2024-02-19T19:01:09.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:09.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46394/0 (socket says 172.21.15.39:46394) 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 -- 172.21.15.39:0/374110194 learned_addr learned my addr 172.21.15.39:0/374110194 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa13659c700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138105bd0 0x7fa138101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 -- 172.21.15.39:0/374110194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138105bd0 msgr2=0x7fa138101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138105bd0 0x7fa138101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 -- 172.21.15.39:0/374110194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c0095f0 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa136d9d700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138101840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c930 tx=0x7fa12800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.726+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa128007ad0 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.727+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1381a2a30 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.727+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa12800ce80 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.727+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa128018630 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.727+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1381a2d90 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.727+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa13810cfa0 con 0x7fa1380ffc20 2024-02-19T19:01:09.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.728+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa128007c30 con 0x7fa1380ffc20 2024-02-19T19:01:09.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.728+0000 7fa11ffff700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa12006c4e0 0x7fa12006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:09.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.728+0000 7fa13659c700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa12006c4e0 0x7fa12006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:09.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.728+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa12808b450 con 0x7fa1380ffc20 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.729+0000 7fa11ffff700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] conn(0x7fa1200721c0 0x7fa1200745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.729+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 --> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa120074c90 con 0x7fa1200721c0 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.729+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa12808db20 con 0x7fa1380ffc20 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.729+0000 7fa13759e700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] conn(0x7fa1200721c0 0x7fa1200745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.729+0000 7fa13659c700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa12006c4e0 0x7fa12006e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa138103030 tx=0x7fa12c01a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.729+0000 7fa13759e700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] conn(0x7fa1200721c0 0x7fa1200745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:09.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.730+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== osd.1 v2:172.21.15.39:6810/3708594483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa120074c90 con 0x7fa1200721c0 2024-02-19T19:01:09.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.746+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 --> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa13804f2e0 con 0x7fa1200721c0 2024-02-19T19:01:09.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.746+0000 7fa11ffff700 1 -- 172.21.15.39:0/374110194 <== osd.1 v2:172.21.15.39:6810/3708594483 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa13804f2e0 con 0x7fa1200721c0 2024-02-19T19:01:09.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] conn(0x7fa1200721c0 msgr2=0x7fa1200745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:09.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] conn(0x7fa1200721c0 0x7fa1200745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa12006c4e0 msgr2=0x7fa12006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa12006c4e0 0x7fa12006e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa138103030 tx=0x7fa12c01a040 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 msgr2=0x7fa138101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138101840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c930 tx=0x7fa12800cc40 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 shutdown_connections 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6810/3708594483,v1:172.21.15.39:6811/3708594483] conn(0x7fa1200721c0 0x7fa1200745e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa12006c4e0 0x7fa12006e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1380ffc20 0x7fa138101840 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 --2- 172.21.15.39:0/374110194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138105bd0 0x7fa138101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 >> 172.21.15.39:0/374110194 conn(0x7fa138075670 msgr2=0x7fa138068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 shutdown_connections 2024-02-19T19:01:09.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:09.747+0000 7fa13d1e3700 1 -- 172.21.15.39:0/374110194 wait complete. 2024-02-19T19:01:09.851 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:09 smithi039 ceph-mon[28988]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:10.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:09 smithi086 ceph-mon[37629]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:10.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 -- 172.21.15.39:0/1606442488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 msgr2=0x7f87601047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:10.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 --2- 172.21.15.39:0/1606442488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f87601047a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8754009910 tx=0x7f8754009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:10.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 -- 172.21.15.39:0/1606442488 shutdown_connections 2024-02-19T19:01:10.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 --2- 172.21.15.39:0/1606442488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f87601047a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 --2- 172.21.15.39:0/1606442488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87600ffda0 0x7f8760100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 -- 172.21.15.39:0/1606442488 >> 172.21.15.39:0/1606442488 conn(0x7f87600fb650 msgr2=0x7f87600fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:10.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 -- 172.21.15.39:0/1606442488 shutdown_connections 2024-02-19T19:01:10.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.831+0000 7f8766271700 1 -- 172.21.15.39:0/1606442488 wait complete. 2024-02-19T19:01:10.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f8766271700 1 Processor -- start 2024-02-19T19:01:10.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f8766271700 1 -- start start 2024-02-19T19:01:10.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f8766271700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87600ffda0 0x7f876019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f8766271700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f876019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f8766271700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876019d240 con 0x7f8760100750 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f8766271700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87601966f0 con 0x7f87600ffda0 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f875f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f876019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.832+0000 7f875f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f876019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46412/0 (socket says 172.21.15.39:46412) 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875f7fe700 1 -- 172.21.15.39:0/3052373676 learned_addr learned my addr 172.21.15.39:0/3052373676 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875ffff700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87600ffda0 0x7f876019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875f7fe700 1 -- 172.21.15.39:0/3052373676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87600ffda0 msgr2=0x7f876019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875f7fe700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87600ffda0 0x7f876019c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875f7fe700 1 -- 172.21.15.39:0/3052373676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87540095f0 con 0x7f8760100750 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875f7fe700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f876019cbb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f875400bda0 tx=0x7f875400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f875401d070 con 0x7f8760100750 2024-02-19T19:01:10.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f875400f460 con 0x7f8760100750 2024-02-19T19:01:10.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8760196970 con 0x7f8760100750 2024-02-19T19:01:10.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8754017630 con 0x7f8760100750 2024-02-19T19:01:10.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.833+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8760196e60 con 0x7f8760100750 2024-02-19T19:01:10.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.834+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f876010cfc0 con 0x7f8760100750 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f875400fab0 con 0x7f8760100750 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875d7fa700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f874806c4e0 0x7f874806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875ffff700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f874806c4e0 0x7f874806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f875408cca0 con 0x7f8760100750 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875d7fa700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] conn(0x7f87480721c0 0x7f87480745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 --> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8748074c90 con 0x7f87480721c0 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.835+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f875408d100 con 0x7f8760100750 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.836+0000 7f876480e700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] conn(0x7f87480721c0 0x7f87480745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:10.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.836+0000 7f875ffff700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f874806c4e0 0x7f874806e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8750007be0 tx=0x7f87500073d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:10.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.836+0000 7f876480e700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] conn(0x7f87480721c0 0x7f87480745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:10.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.837+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== osd.2 v2:172.21.15.39:6818/3928116015 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8748074c90 con 0x7f87480721c0 2024-02-19T19:01:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.853+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 --> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f876004f2e0 con 0x7f87480721c0 2024-02-19T19:01:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.853+0000 7f875d7fa700 1 -- 172.21.15.39:0/3052373676 <== osd.2 v2:172.21.15.39:6818/3928116015 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f876004f2e0 con 0x7f87480721c0 2024-02-19T19:01:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.853+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] conn(0x7f87480721c0 msgr2=0x7f87480745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.853+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] conn(0x7f87480721c0 0x7f87480745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.853+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f874806c4e0 msgr2=0x7f874806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.853+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f874806c4e0 0x7f874806e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8750007be0 tx=0x7f87500073d0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 msgr2=0x7f876019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f876019cbb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f875400bda0 tx=0x7f875400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 shutdown_connections 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6818/3928116015,v1:172.21.15.39:6819/3928116015] conn(0x7f87480721c0 0x7f87480745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f874806c4e0 0x7f874806e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87600ffda0 0x7f876019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 --2- 172.21.15.39:0/3052373676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8760100750 0x7f876019cbb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 >> 172.21.15.39:0/3052373676 conn(0x7f87600fb650 msgr2=0x7f8760069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 shutdown_connections 2024-02-19T19:01:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:10.854+0000 7f8766271700 1 -- 172.21.15.39:0/3052373676 wait complete. 2024-02-19T19:01:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.350+0000 7fa0fd1da700 1 -- 172.21.15.39:0/3689626100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 msgr2=0x7fa0f81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.350+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/3689626100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f81047a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009910 tx=0x7fa0ec009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.350+0000 7fa0fd1da700 1 -- 172.21.15.39:0/3689626100 shutdown_connections 2024-02-19T19:01:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/3689626100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f81047a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/3689626100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0f80ffda0 0x7fa0f8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 -- 172.21.15.39:0/3689626100 >> 172.21.15.39:0/3689626100 conn(0x7fa0f80fb650 msgr2=0x7fa0f80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:11.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 -- 172.21.15.39:0/3689626100 shutdown_connections 2024-02-19T19:01:11.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 -- 172.21.15.39:0/3689626100 wait complete. 2024-02-19T19:01:11.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 Processor -- start 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.351+0000 7fa0fd1da700 1 -- start start 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0fd1da700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0f80ffda0 0x7fa0f819b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0fd1da700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0fd1da700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f819c1f0 con 0x7fa0f8100750 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0fd1da700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f81956a0 con 0x7fa0f80ffda0 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f819bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46430/0 (socket says 172.21.15.39:46430) 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 -- 172.21.15.39:0/1676039340 learned_addr learned my addr 172.21.15.39:0/1676039340 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f6d9d700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0f80ffda0 0x7fa0f819b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 -- 172.21.15.39:0/1676039340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0f80ffda0 msgr2=0x7fa0f819b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0f80ffda0 0x7fa0f819b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 -- 172.21.15.39:0/1676039340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec0095f0 con 0x7fa0f8100750 2024-02-19T19:01:11.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0f659c700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f819bb60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009fd0 tx=0x7fa0ec00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:11.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0ec01d070 con 0x7fa0f8100750 2024-02-19T19:01:11.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f8195920 con 0x7fa0f8100750 2024-02-19T19:01:11.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.352+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa0ec005640 con 0x7fa0f8100750 2024-02-19T19:01:11.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.353+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0ec0226a0 con 0x7fa0f8100750 2024-02-19T19:01:11.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.353+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f8195e10 con 0x7fa0f8100750 2024-02-19T19:01:11.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.353+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0f810cfc0 con 0x7fa0f8100750 2024-02-19T19:01:11.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.354+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa0ec003680 con 0x7fa0f8100750 2024-02-19T19:01:11.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.354+0000 7fa0dffff700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa0e006c4e0 0x7fa0e006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:11.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.354+0000 7fa0f6d9d700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa0e006c4e0 0x7fa0e006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:11.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.354+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa0ec08c970 con 0x7fa0f8100750 2024-02-19T19:01:11.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.355+0000 7fa0dffff700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] conn(0x7fa0e00721c0 0x7fa0e00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:11.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.355+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 --> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0e0074c90 con 0x7fa0e00721c0 2024-02-19T19:01:11.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.355+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0ec0912a0 con 0x7fa0f8100750 2024-02-19T19:01:11.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.355+0000 7fa0f759e700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] conn(0x7fa0e00721c0 0x7fa0e00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:11.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.355+0000 7fa0f6d9d700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa0e006c4e0 0x7fa0e006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009bf0 tx=0x7fa0e8009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:11.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.355+0000 7fa0f759e700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] conn(0x7fa0e00721c0 0x7fa0e00745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:11.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.356+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== osd.0 v2:172.21.15.39:6802/2663416145 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa0e0074c90 con 0x7fa0e00721c0 2024-02-19T19:01:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.371+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 --> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa0f804f2e0 con 0x7fa0e00721c0 2024-02-19T19:01:11.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.371+0000 7fa0dffff700 1 -- 172.21.15.39:0/1676039340 <== osd.0 v2:172.21.15.39:6802/2663416145 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa0f804f2e0 con 0x7fa0e00721c0 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] conn(0x7fa0e00721c0 msgr2=0x7fa0e00745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] conn(0x7fa0e00721c0 0x7fa0e00745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa0e006c4e0 msgr2=0x7fa0e006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa0e006c4e0 0x7fa0e006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009bf0 tx=0x7fa0e8009260 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 msgr2=0x7fa0f819bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f819bb60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec009fd0 tx=0x7fa0ec00fab0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 shutdown_connections 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6802/2663416145,v1:172.21.15.39:6803/2663416145] conn(0x7fa0e00721c0 0x7fa0e00745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa0e006c4e0 0x7fa0e006e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0f80ffda0 0x7fa0f819b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 --2- 172.21.15.39:0/1676039340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0f8100750 0x7fa0f819bb60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 >> 172.21.15.39:0/1676039340 conn(0x7fa0f80fb650 msgr2=0x7fa0f80fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 shutdown_connections 2024-02-19T19:01:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:11.372+0000 7fa0fd1da700 1 -- 172.21.15.39:0/1676039340 wait complete. 2024-02-19T19:01:12.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:11 smithi039 ceph-mon[28988]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:12.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:11 smithi086 ceph-mon[37629]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:12.623 INFO:teuthology.orchestra.run.smithi039.stdout:81604378648 2024-02-19T19:01:12.623 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-19T19:01:13.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 -- 172.21.15.39:0/2996013483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 msgr2=0x7f1fe41047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/2996013483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe41047b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009910 tx=0x7f1fd8009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:13.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 -- 172.21.15.39:0/2996013483 shutdown_connections 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/2996013483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe41047b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/2996013483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe41006b0 0x7f1fe4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 -- 172.21.15.39:0/2996013483 >> 172.21.15.39:0/2996013483 conn(0x7f1fe4074e30 msgr2=0x7f1fe4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 -- 172.21.15.39:0/2996013483 shutdown_connections 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.265+0000 7f1fea4a7700 1 -- 172.21.15.39:0/2996013483 wait complete. 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fea4a7700 1 Processor -- start 2024-02-19T19:01:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fea4a7700 1 -- start start 2024-02-19T19:01:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fea4a7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe41006b0 0x7f1fe41977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fea4a7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe4197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fea4a7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4198300 con 0x7f1fe4101060 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fea4a7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4198440 con 0x7f1fe41006b0 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.266+0000 7f1fe37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe4197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe4197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46450/0 (socket says 172.21.15.39:46450) 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe37fe700 1 -- 172.21.15.39:0/1693932691 learned_addr learned my addr 172.21.15.39:0/1693932691 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe3fff700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe41006b0 0x7f1fe41977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe37fe700 1 -- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe41006b0 msgr2=0x7f1fe41977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe37fe700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe41006b0 0x7f1fe41977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe37fe700 1 -- 172.21.15.39:0/1693932691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80095f0 con 0x7f1fe4101060 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe37fe700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe4197ce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800bda0 tx=0x7f1fd800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd801d070 con 0x7f1fe4101060 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1fd8004fa0 con 0x7f1fe4101060 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd800f690 con 0x7f1fe4101060 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe41a0810 con 0x7f1fe4101060 2024-02-19T19:01:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.267+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe41a0c20 con 0x7f1fe4101060 2024-02-19T19:01:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.268+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1fe410d050 con 0x7f1fe4101060 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1fd80228c0 con 0x7f1fe4101060 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe17fa700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fcc06c4e0 0x7f1fcc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1fd808cc60 con 0x7f1fe4101060 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe3fff700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fcc06c4e0 0x7f1fcc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe17fa700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] conn(0x7f1fcc0721c0 0x7f1fcc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 --> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fcc074c90 con 0x7f1fcc0721c0 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.269+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd8092340 con 0x7f1fe4101060 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.270+0000 7f1fe8a44700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] conn(0x7f1fcc0721c0 0x7f1fcc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.270+0000 7f1fe3fff700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fcc06c4e0 0x7f1fcc06e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009a10 tx=0x7f1fd4009190 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.270+0000 7f1fe8a44700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] conn(0x7f1fcc0721c0 0x7f1fcc0745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.271+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== osd.4 v2:172.21.15.86:6808/2218646412 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1fcc074c90 con 0x7f1fcc0721c0 2024-02-19T19:01:13.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.286+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 --> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1fe404f2e0 con 0x7f1fcc0721c0 2024-02-19T19:01:13.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fe17fa700 1 -- 172.21.15.39:0/1693932691 <== osd.4 v2:172.21.15.86:6808/2218646412 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1fe404f2e0 con 0x7f1fcc0721c0 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] conn(0x7f1fcc0721c0 msgr2=0x7f1fcc0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] conn(0x7f1fcc0721c0 0x7f1fcc0745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fcc06c4e0 msgr2=0x7f1fcc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fcc06c4e0 0x7f1fcc06e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009a10 tx=0x7f1fd4009190 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 msgr2=0x7f1fe4197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe4197ce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800bda0 tx=0x7f1fd800bdd0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 shutdown_connections 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fcc06c4e0 0x7f1fcc06e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe41006b0 0x7f1fe41977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.86:6808/2218646412,v1:172.21.15.86:6809/2218646412] conn(0x7f1fcc0721c0 0x7f1fcc0745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 --2- 172.21.15.39:0/1693932691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fe4101060 0x7f1fe4197ce0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 >> 172.21.15.39:0/1693932691 conn(0x7f1fe4074e30 msgr2=0x7f1fe40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 shutdown_connections 2024-02-19T19:01:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.287+0000 7f1fea4a7700 1 -- 172.21.15.39:0/1693932691 wait complete. 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 -- 172.21.15.39:0/274488638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 msgr2=0x7fdb941021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 --2- 172.21.15.39:0/274488638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb941021e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009910 tx=0x7fdb84009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 -- 172.21.15.39:0/274488638 shutdown_connections 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 --2- 172.21.15.39:0/274488638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb94102720 0x7fdb9410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 --2- 172.21.15.39:0/274488638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb941021e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 -- 172.21.15.39:0/274488638 >> 172.21.15.39:0/274488638 conn(0x7fdb940fb650 msgr2=0x7fdb940fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 -- 172.21.15.39:0/274488638 shutdown_connections 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 -- 172.21.15.39:0/274488638 wait complete. 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.491+0000 7fdb99719700 1 Processor -- start 2024-02-19T19:01:13.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb99719700 1 -- start start 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb99719700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb9419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb99719700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb94102720 0x7fdb9419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb99719700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9419d2b0 con 0x7fdb94101e00 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb99719700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb941967b0 con 0x7fdb94102720 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb9419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb9419c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46482/0 (socket says 172.21.15.39:46482) 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 -- 172.21.15.39:0/2622579849 learned_addr learned my addr 172.21.15.39:0/2622579849 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb927fc700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb94102720 0x7fdb9419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 -- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb94102720 msgr2=0x7fdb9419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb94102720 0x7fdb9419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 -- 172.21.15.39:0/2622579849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb840095f0 con 0x7fdb94101e00 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.492+0000 7fdb92ffd700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb9419c6e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb84005b40 tx=0x7fdb8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.493+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb8401d070 con 0x7fdb94101e00 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.493+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdb840048e0 con 0x7fdb94101e00 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.493+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb94196a30 con 0x7fdb94101e00 2024-02-19T19:01:13.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.493+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb94196ec0 con 0x7fdb94101e00 2024-02-19T19:01:13.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.493+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb84003b90 con 0x7fdb94101e00 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.494+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb9410e170 con 0x7fdb94101e00 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.494+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdb84004a50 con 0x7fdb94101e00 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb7bfff700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb7c06c4e0 0x7fdb7c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdb8408cc90 con 0x7fdb94101e00 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb927fc700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb7c06c4e0 0x7fdb7c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb7bfff700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] conn(0x7fdb7c0721c0 0x7fdb7c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 --> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb7c074c90 con 0x7fdb7c0721c0 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb8408d0f0 con 0x7fdb94101e00 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb927fc700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb7c06c4e0 0x7fdb7c06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb94197bd0 tx=0x7fdb88009190 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.495+0000 7fdb937fe700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] conn(0x7fdb7c0721c0 0x7fdb7c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.496+0000 7fdb937fe700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] conn(0x7fdb7c0721c0 0x7fdb7c0745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:13.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.496+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== osd.5 v2:172.21.15.86:6816/1712053993 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdb7c074c90 con 0x7fdb7c0721c0 2024-02-19T19:01:13.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.512+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 --> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb9404f2e0 con 0x7fdb7c0721c0 2024-02-19T19:01:13.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb7bfff700 1 -- 172.21.15.39:0/2622579849 <== osd.5 v2:172.21.15.86:6816/1712053993 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdb9404f2e0 con 0x7fdb7c0721c0 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] conn(0x7fdb7c0721c0 msgr2=0x7fdb7c0745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] conn(0x7fdb7c0721c0 0x7fdb7c0745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb7c06c4e0 msgr2=0x7fdb7c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb7c06c4e0 0x7fdb7c06e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb94197bd0 tx=0x7fdb88009190 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 msgr2=0x7fdb9419c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb9419c6e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb84005b40 tx=0x7fdb8400bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 shutdown_connections 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb7c06c4e0 0x7fdb7c06e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:6816/1712053993,v1:172.21.15.86:6817/1712053993] conn(0x7fdb7c0721c0 0x7fdb7c0745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb94101e00 0x7fdb9419c6e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 --2- 172.21.15.39:0/2622579849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb94102720 0x7fdb9419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 >> 172.21.15.39:0/2622579849 conn(0x7fdb940fb650 msgr2=0x7fdb94105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 shutdown_connections 2024-02-19T19:01:13.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:13.513+0000 7fdb99719700 1 -- 172.21.15.39:0/2622579849 wait complete. 2024-02-19T19:01:13.898 INFO:teuthology.orchestra.run.smithi039.stdout:60129542173 2024-02-19T19:01:13.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-19T19:01:14.146 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:13 smithi039 ceph-mon[28988]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:14.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:13 smithi086 ceph-mon[37629]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:15.033 INFO:teuthology.orchestra.run.smithi039.stdout:167503724553 2024-02-19T19:01:15.033 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-19T19:01:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:14 smithi039 ceph-mon[28988]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:15.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:14 smithi086 ceph-mon[37629]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:15.431 INFO:teuthology.orchestra.run.smithi039.stdout:38654705699 2024-02-19T19:01:15.432 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-19T19:01:16.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:16.320 INFO:teuthology.orchestra.run.smithi039.stdout:146028888079 2024-02-19T19:01:16.320 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-19T19:01:16.344 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:16.345 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2436614768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb40ffb40 msgr2=0x7f1fb40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2436614768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb40ffb40 0x7f1fb40fffc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009910 tx=0x7f1fa4009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2436614768 shutdown_connections 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2436614768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb40ffb40 0x7f1fb40fffc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2436614768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2436614768 >> 172.21.15.39:0/2436614768 conn(0x7f1fb40fb610 msgr2=0x7f1fb40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2436614768 shutdown_connections 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2436614768 wait complete. 2024-02-19T19:01:16.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 Processor -- start 2024-02-19T19:01:16.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.955+0000 7f1fbb93d700 1 -- start start 2024-02-19T19:01:16.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fbb93d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb40ffb40 0x7f1fb4197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:16.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fbb93d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:16.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fbb93d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb4198240 con 0x7f1fb4105bf0 2024-02-19T19:01:16.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fbb93d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb4198380 con 0x7f1fb40ffb40 2024-02-19T19:01:16.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:16.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37086/0 (socket says 172.21.15.39:37086) 2024-02-19T19:01:16.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 -- 172.21.15.39:0/2700205691 learned_addr learned my addr 172.21.15.39:0/2700205691 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:16.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb96d9700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb40ffb40 0x7f1fb4197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:16.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 -- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb40ffb40 msgr2=0x7f1fb4197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:16.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb40ffb40 0x7f1fb4197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 -- 172.21.15.39:0/2700205691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40095f0 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb96d9700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb40ffb40 0x7f1fb4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1fb8ed8700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4197b60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4006010 tx=0x7f1fa400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.956+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fa401d070 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.957+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb41a0690 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.957+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1fa400f460 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.957+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fa4021640 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.957+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb41a0b00 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.957+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1fb410cff0 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.958+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1fa400faf0 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.958+0000 7f1faa7fc700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fa006c4e0 0x7f1fa006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.958+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1fa408d3a0 con 0x7f1fb4105bf0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.958+0000 7f1fb96d9700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fa006c4e0 0x7f1fa006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:16.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.959+0000 7f1faa7fc700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] conn(0x7f1fa00721c0 0x7f1fa00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:16.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.959+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 --> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fa0074c90 con 0x7f1fa00721c0 2024-02-19T19:01:16.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.959+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fa408d800 con 0x7f1fb4105bf0 2024-02-19T19:01:16.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.959+0000 7f1fb96d9700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fa006c4e0 0x7f1fa006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0005fd0 tx=0x7f1fb0005dc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:16.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.959+0000 7f1fb9eda700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] conn(0x7f1fa00721c0 0x7f1fa00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:16.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.959+0000 7f1fb9eda700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] conn(0x7f1fa00721c0 0x7f1fa00745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:16.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.960+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== osd.3 v2:172.21.15.86:6800/1886206958 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1fa0074c90 con 0x7f1fa00721c0 2024-02-19T19:01:16.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.979+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 --> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1fb40679c0 con 0x7f1fa00721c0 2024-02-19T19:01:16.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1faa7fc700 1 -- 172.21.15.39:0/2700205691 <== osd.3 v2:172.21.15.86:6800/1886206958 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1fb40679c0 con 0x7f1fa00721c0 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] conn(0x7f1fa00721c0 msgr2=0x7f1fa00745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] conn(0x7f1fa00721c0 0x7f1fa00745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fa006c4e0 msgr2=0x7f1fa006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fa006c4e0 0x7f1fa006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0005fd0 tx=0x7f1fb0005dc0 comp rx=0 tx=0).stop 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 msgr2=0x7f1fb4197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4197b60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4006010 tx=0x7f1fa400be20 comp rx=0 tx=0).stop 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 shutdown_connections 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:6800/1886206958,v1:172.21.15.86:6801/1886206958] conn(0x7f1fa00721c0 0x7f1fa00745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1fa006c4e0 0x7f1fa006e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fb40ffb40 0x7f1fb4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 --2- 172.21.15.39:0/2700205691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb4105bf0 0x7f1fb4197b60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:17.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 >> 172.21.15.39:0/2700205691 conn(0x7f1fb40fb610 msgr2=0x7f1fb40fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:17.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 shutdown_connections 2024-02-19T19:01:17.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:16.980+0000 7f1fbb93d700 1 -- 172.21.15.39:0/2700205691 wait complete. 2024-02-19T19:01:17.164 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:16 smithi039 ceph-mon[28988]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:17.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:16 smithi086 ceph-mon[37629]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:18.595 INFO:teuthology.orchestra.run.smithi039.stdout:111669149716 2024-02-19T19:01:18.596 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-19T19:01:19.820 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:19.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.811+0000 7f511a21b700 1 -- 172.21.15.39:0/1930647760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140fff50 msgr2=0x7f5114105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:19.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.811+0000 7f511a21b700 1 --2- 172.21.15.39:0/1930647760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140fff50 0x7f5114105840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5108009910 tx=0x7f5108009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:19.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.812+0000 7f511a21b700 1 -- 172.21.15.39:0/1930647760 shutdown_connections 2024-02-19T19:01:19.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.812+0000 7f511a21b700 1 --2- 172.21.15.39:0/1930647760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140fff50 0x7f5114105840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:19.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.812+0000 7f511a21b700 1 --2- 172.21.15.39:0/1930647760 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140ff5a0 0x7f51140ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:19.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.812+0000 7f511a21b700 1 -- 172.21.15.39:0/1930647760 >> 172.21.15.39:0/1930647760 conn(0x7f5114074e30 msgr2=0x7f5114075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:19.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.812+0000 7f511a21b700 1 -- 172.21.15.39:0/1930647760 shutdown_connections 2024-02-19T19:01:19.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.812+0000 7f511a21b700 1 -- 172.21.15.39:0/1930647760 wait complete. 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.813+0000 7f511a21b700 1 Processor -- start 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.813+0000 7f511a21b700 1 -- start start 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.813+0000 7f511a21b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 0x7f5114197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.813+0000 7f511a21b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140fff50 0x7f5114194540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.813+0000 7f511a21b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114194b10 con 0x7f51140ff5a0 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.813+0000 7f511a21b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5114194c80 con 0x7f51140fff50 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f51137fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 0x7f5114197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f51137fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 0x7f5114197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37102/0 (socket says 172.21.15.39:37102) 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f51137fe700 1 -- 172.21.15.39:0/394849150 learned_addr learned my addr 172.21.15.39:0/394849150 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f51137fe700 1 -- 172.21.15.39:0/394849150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140fff50 msgr2=0x7f5114194540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f5112ffd700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140fff50 0x7f5114194540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:19.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f51137fe700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140fff50 0x7f5114194540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f51137fe700 1 -- 172.21.15.39:0/394849150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51080095f0 con 0x7f51140ff5a0 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.814+0000 7f5112ffd700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140fff50 0x7f5114194540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.815+0000 7f51137fe700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 0x7f5114197310 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f510400b4b0 tx=0x7f510400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.815+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5104011800 con 0x7f51140ff5a0 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.815+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5104011e10 con 0x7f51140ff5a0 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.815+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5114194f60 con 0x7f51140ff5a0 2024-02-19T19:01:19.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.815+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51141a7190 con 0x7f51140ff5a0 2024-02-19T19:01:19.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.815+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510400f2e0 con 0x7f51140ff5a0 2024-02-19T19:01:19.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.816+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5100005320 con 0x7f51140ff5a0 2024-02-19T19:01:19.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.817+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f510400f440 con 0x7f51140ff5a0 2024-02-19T19:01:19.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.817+0000 7f5110ff9700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f50fc06c4e0 0x7f50fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:19.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.818+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f510408afa0 con 0x7f51140ff5a0 2024-02-19T19:01:19.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.818+0000 7f5112ffd700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f50fc06c4e0 0x7f50fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:19.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.818+0000 7f5112ffd700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f50fc06c4e0 0x7f50fc06e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5114195b20 tx=0x7f5108005960 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:19.839 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:19.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:19.827+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f51040be0e0 con 0x7f51140ff5a0 2024-02-19T19:01:19.873 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:19.980 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:19 smithi039 ceph-mon[28988]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:20.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:19 smithi086 ceph-mon[37629]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:20.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.049+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5100005190 con 0x7f51140ff5a0 2024-02-19T19:01:20.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.049+0000 7f5110ff9700 1 -- 172.21.15.39:0/394849150 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5104015090 con 0x7f51140ff5a0 2024-02-19T19:01:20.068 INFO:teuthology.orchestra.run.smithi039.stdout:60129542175 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f50fc06c4e0 msgr2=0x7f50fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f50fc06c4e0 0x7f50fc06e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5114195b20 tx=0x7f5108005960 comp rx=0 tx=0).stop 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 msgr2=0x7f5114197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 0x7f5114197310 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f510400b4b0 tx=0x7f510400b870 comp rx=0 tx=0).stop 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 shutdown_connections 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f50fc06c4e0 0x7f50fc06e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51140ff5a0 0x7f5114197310 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:20.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 --2- 172.21.15.39:0/394849150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51140fff50 0x7f5114194540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:20.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.051+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 >> 172.21.15.39:0/394849150 conn(0x7f5114074e30 msgr2=0x7f51140fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:20.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.052+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 shutdown_connections 2024-02-19T19:01:20.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:20.052+0000 7f511a21b700 1 -- 172.21.15.39:0/394849150 wait complete. 2024-02-19T19:01:20.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:20 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/394849150' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T19:01:21.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:20 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/394849150' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T19:01:21.352 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:21.910 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:21 smithi039 ceph-mon[28988]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:21.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542175 for osd.1 2024-02-19T19:01:21.932 DEBUG:teuthology.parallel:result is None 2024-02-19T19:01:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:21 smithi086 ceph-mon[37629]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 -- 172.21.15.39:0/2803017037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 msgr2=0x7fd880105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2803017037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd880105840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd874009910 tx=0x7fd874009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 -- 172.21.15.39:0/2803017037 shutdown_connections 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2803017037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd880105840 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2803017037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8800ff5a0 0x7fd8800ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 -- 172.21.15.39:0/2803017037 >> 172.21.15.39:0/2803017037 conn(0x7fd880074e30 msgr2=0x7fd880075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 -- 172.21.15.39:0/2803017037 shutdown_connections 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.421+0000 7fd885cd4700 1 -- 172.21.15.39:0/2803017037 wait complete. 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.422+0000 7fd885cd4700 1 Processor -- start 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.422+0000 7fd885cd4700 1 -- start start 2024-02-19T19:01:23.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.422+0000 7fd885cd4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8800ff5a0 0x7fd88019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.422+0000 7fd885cd4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd88019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.422+0000 7fd885cd4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88019f420 con 0x7fd8800fff50 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.422+0000 7fd885cd4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8801988d0 con 0x7fd8800ff5a0 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd88019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd88019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37126/0 (socket says 172.21.15.39:37126) 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 -- 172.21.15.39:0/2744236918 learned_addr learned my addr 172.21.15.39:0/2744236918 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87f7fe700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8800ff5a0 0x7fd88019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 -- 172.21.15.39:0/2744236918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8800ff5a0 msgr2=0x7fd88019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8800ff5a0 0x7fd88019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 -- 172.21.15.39:0/2744236918 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740095f0 con 0x7fd8800fff50 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87effd700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd88019ed90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8740092e0 tx=0x7fd87400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:23.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd87401d070 con 0x7fd8800fff50 2024-02-19T19:01:23.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd874005dc0 con 0x7fd8800fff50 2024-02-19T19:01:23.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd874017400 con 0x7fd8800fff50 2024-02-19T19:01:23.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd880198b50 con 0x7fd8800fff50 2024-02-19T19:01:23.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.423+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd880198ff0 con 0x7fd8800fff50 2024-02-19T19:01:23.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.424+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd880104fc0 con 0x7fd8800fff50 2024-02-19T19:01:23.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.425+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd87400f460 con 0x7fd8800fff50 2024-02-19T19:01:23.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.425+0000 7fd87cff9700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd86806c4e0 0x7fd86806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:23.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.425+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd87408c950 con 0x7fd8800fff50 2024-02-19T19:01:23.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.425+0000 7fd87f7fe700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd86806c4e0 0x7fd86806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:23.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.426+0000 7fd87f7fe700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd86806c4e0 0x7fd86806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd870005950 tx=0x7fd87000b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:23.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.429+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd87405af00 con 0x7fd8800fff50 2024-02-19T19:01:23.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.628+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd8801049a0 con 0x7fd8800fff50 2024-02-19T19:01:23.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.629+0000 7fd87cff9700 1 -- 172.21.15.39:0/2744236918 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd87405aa90 con 0x7fd8800fff50 2024-02-19T19:01:23.647 INFO:teuthology.orchestra.run.smithi039.stdout:167503724555 2024-02-19T19:01:23.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd86806c4e0 msgr2=0x7fd86806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:23.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd86806c4e0 0x7fd86806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd870005950 tx=0x7fd87000b410 comp rx=0 tx=0).stop 2024-02-19T19:01:23.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 msgr2=0x7fd88019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd88019ed90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd8740092e0 tx=0x7fd87400bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 shutdown_connections 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd86806c4e0 0x7fd86806e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8800ff5a0 0x7fd88019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 --2- 172.21.15.39:0/2744236918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8800fff50 0x7fd88019ed90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 >> 172.21.15.39:0/2744236918 conn(0x7fd880074e30 msgr2=0x7fd8800fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 shutdown_connections 2024-02-19T19:01:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:23.631+0000 7fd885cd4700 1 -- 172.21.15.39:0/2744236918 wait complete. 2024-02-19T19:01:23.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:23 smithi039 ceph-mon[28988]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:23.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:23 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2744236918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T19:01:24.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:23 smithi086 ceph-mon[37629]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:24.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:23 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2744236918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T19:01:25.072 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:24 smithi039 ceph-mon[28988]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:24 smithi086 ceph-mon[37629]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:25.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 -- 172.21.15.39:0/4105356120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 msgr2=0x7f7c700ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:25.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/4105356120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c700ff9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000b0a0 tx=0x7f7c6000b3b0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 -- 172.21.15.39:0/4105356120 shutdown_connections 2024-02-19T19:01:25.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/4105356120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c700fffb0 0x7f7c701058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/4105356120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c700ff9e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 -- 172.21.15.39:0/4105356120 >> 172.21.15.39:0/4105356120 conn(0x7f7c70074df0 msgr2=0x7f7c70075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:25.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 -- 172.21.15.39:0/4105356120 shutdown_connections 2024-02-19T19:01:25.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 -- 172.21.15.39:0/4105356120 wait complete. 2024-02-19T19:01:25.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 Processor -- start 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.593+0000 7f7c74fd8700 1 -- start start 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c74fd8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c7019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c74fd8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c700fffb0 0x7f7c7019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c74fd8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7019d2c0 con 0x7f7c700ff600 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c74fd8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70196770 con 0x7f7c700fffb0 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c7019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c7019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37158/0 (socket says 172.21.15.39:37158) 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 -- 172.21.15.39:0/682619556 learned_addr learned my addr 172.21.15.39:0/682619556 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6dd9b700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c700fffb0 0x7f7c7019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 -- 172.21.15.39:0/682619556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c700fffb0 msgr2=0x7f7c7019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c700fffb0 0x7f7c7019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 -- 172.21.15.39:0/682619556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009d00 con 0x7f7c700ff600 2024-02-19T19:01:25.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c6e59c700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c7019c6f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7c60000f80 tx=0x7f7c6000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c6000e040 con 0x7f7c700ff600 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7c60003de0 con 0x7f7c700ff600 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.594+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c701969f0 con 0x7f7c700ff600 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.595+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c6001d840 con 0x7f7c700ff600 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.595+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70196ee0 con 0x7f7c700ff600 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.595+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70109270 con 0x7f7c700ff600 2024-02-19T19:01:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.596+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7c60004040 con 0x7f7c700ff600 2024-02-19T19:01:25.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.596+0000 7f7c5f7fe700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:25.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.596+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7c6008cd50 con 0x7f7c700ff600 2024-02-19T19:01:25.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.596+0000 7f7c6dd9b700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:25.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.597+0000 7f7c6dd9b700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7c70197be0 tx=0x7f7c640073d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:25.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.601+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7c6005b300 con 0x7f7c700ff600 2024-02-19T19:01:25.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.800+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7c7004f2e0 con 0x7f7c700ff600 2024-02-19T19:01:25.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.801+0000 7f7c5f7fe700 1 -- 172.21.15.39:0/682619556 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7c6005ae90 con 0x7f7c700ff600 2024-02-19T19:01:25.819 INFO:teuthology.orchestra.run.smithi039.stdout:146028888081 2024-02-19T19:01:25.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 -- 172.21.15.39:0/727659720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d4105bd0 msgr2=0x7fb6d4105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:25.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/727659720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d4105bd0 0x7fb6d4105fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009910 tx=0x7fb6d0009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:25.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 -- 172.21.15.39:0/727659720 shutdown_connections 2024-02-19T19:01:25.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/727659720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6d40ffc20 0x7fb6d4100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/727659720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d4105bd0 0x7fb6d4105fb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 -- 172.21.15.39:0/727659720 >> 172.21.15.39:0/727659720 conn(0x7fb6d4075670 msgr2=0x7fb6d4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:25.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 -- 172.21.15.39:0/727659720 shutdown_connections 2024-02-19T19:01:25.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.802+0000 7fb6dc24c700 1 -- 172.21.15.39:0/727659720 wait complete. 2024-02-19T19:01:25.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6dc24c700 1 Processor -- start 2024-02-19T19:01:25.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c5806c4e0 msgr2=0x7f7c5806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:25.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7c70197be0 tx=0x7f7c640073d0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 msgr2=0x7f7c7019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:25.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c7019c6f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7c60000f80 tx=0x7f7c6000cd20 comp rx=0 tx=0).stop 2024-02-19T19:01:25.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6dc24c700 1 -- start start 2024-02-19T19:01:25.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6dc24c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 0x7fb6d41976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:25.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6d9fe8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 0x7fb6d41976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 shutdown_connections 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c5806c4e0 0x7f7c5806e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c700ff600 0x7f7c7019c6f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 --2- 172.21.15.39:0/682619556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c700fffb0 0x7f7c7019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 >> 172.21.15.39:0/682619556 conn(0x7f7c70074df0 msgr2=0x7f7c700fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6d9fe8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 0x7fb6d41976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37180/0 (socket says 172.21.15.39:37180) 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6d9fe8700 1 -- 172.21.15.39:0/812822066 learned_addr learned my addr 172.21.15.39:0/812822066 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:25.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6d4105bd0 0x7fb6d4197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:25.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d41982f0 con 0x7fb6d40ffc20 2024-02-19T19:01:25.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d4198460 con 0x7fb6d4105bd0 2024-02-19T19:01:25.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 shutdown_connections 2024-02-19T19:01:25.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7f7c74fd8700 1 -- 172.21.15.39:0/682619556 wait complete. 2024-02-19T19:01:25.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.803+0000 7fb6d97e7700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6d4105bd0 0x7fb6d4197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6d9fe8700 1 -- 172.21.15.39:0/812822066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6d4105bd0 msgr2=0x7fb6d4197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6d9fe8700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6d4105bd0 0x7fb6d4197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6d9fe8700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d00095f0 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6d9fe8700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 0x7fb6d41976d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000fa00 tx=0x7fb6d000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6d001d070 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d41a0740 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb6d000fe90 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6d0022470 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.804+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d41a0bb0 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.805+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d41080f0 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.805+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb6d0003680 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.806+0000 7fb6caffd700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb6c006c4e0 0x7fb6c006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.806+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb6d008c8a0 con 0x7fb6d40ffc20 2024-02-19T19:01:25.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.806+0000 7fb6d97e7700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb6c006c4e0 0x7fb6c006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:25.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.806+0000 7fb6d97e7700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb6c006c4e0 0x7fb6c006e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4068e30 tx=0x7fb6c4009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:25.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:25.810+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb6d00573c0 con 0x7fb6d40ffc20 2024-02-19T19:01:26.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.012+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb6d4068c10 con 0x7fb6d40ffc20 2024-02-19T19:01:26.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.012+0000 7fb6caffd700 1 -- 172.21.15.39:0/812822066 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb6d005a9e0 con 0x7fb6d40ffc20 2024-02-19T19:01:26.031 INFO:teuthology.orchestra.run.smithi039.stdout:38654705702 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb6c006c4e0 msgr2=0x7fb6c006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb6c006c4e0 0x7fb6c006e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4068e30 tx=0x7fb6c4009260 comp rx=0 tx=0).stop 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 msgr2=0x7fb6d41976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 0x7fb6d41976d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb6d000fa00 tx=0x7fb6d000fa30 comp rx=0 tx=0).stop 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 shutdown_connections 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb6c006c4e0 0x7fb6c006e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:26.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6d40ffc20 0x7fb6d41976d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:26.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 --2- 172.21.15.39:0/812822066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6d4105bd0 0x7fb6d4197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:26.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 >> 172.21.15.39:0/812822066 conn(0x7fb6d4075670 msgr2=0x7fb6d40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:26.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 shutdown_connections 2024-02-19T19:01:26.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:26.014+0000 7fb6dc24c700 1 -- 172.21.15.39:0/812822066 wait complete. 2024-02-19T19:01:26.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:25 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/682619556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T19:01:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:25 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/682619556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T19:01:27.112 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:26 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/812822066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T19:01:27.112 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:26 smithi039 ceph-mon[28988]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:27.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-02-19T19:01:27.261 DEBUG:teuthology.parallel:result is None 2024-02-19T19:01:27.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:26 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/812822066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T19:01:27.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:26 smithi086 ceph-mon[37629]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:28.074 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-02-19T19:01:28.074 DEBUG:teuthology.parallel:result is None 2024-02-19T19:01:28.705 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724555 for osd.5 2024-02-19T19:01:28.705 DEBUG:teuthology.parallel:result is None 2024-02-19T19:01:28.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1583923674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 msgr2=0x7f7c540ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:28.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1583923674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c540ff890 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009910 tx=0x7f7c3c009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:28.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1583923674 shutdown_connections 2024-02-19T19:01:28.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1583923674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c540ff890 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:28.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1583923674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c54106db0 0x7f7c54107190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:28.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1583923674 >> 172.21.15.39:0/1583923674 conn(0x7f7c54074e30 msgr2=0x7f7c54075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:28.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1583923674 shutdown_connections 2024-02-19T19:01:28.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.912+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1583923674 wait complete. 2024-02-19T19:01:28.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c5a46d700 1 Processor -- start 2024-02-19T19:01:28.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c5a46d700 1 -- start start 2024-02-19T19:01:28.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c5a46d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c5419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c5a46d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c54106db0 0x7f7c5419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c5a46d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5419d2f0 con 0x7f7c540ff410 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c5a46d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c54196770 con 0x7f7c54106db0 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c53fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c5419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c53fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c5419c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42724/0 (socket says 172.21.15.39:42724) 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c53fff700 1 -- 172.21.15.39:0/1007293429 learned_addr learned my addr 172.21.15.39:0/1007293429 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.913+0000 7f7c537fe700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c54106db0 0x7f7c5419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c53fff700 1 -- 172.21.15.39:0/1007293429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c54106db0 msgr2=0x7f7c5419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c53fff700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c54106db0 0x7f7c5419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c53fff700 1 -- 172.21.15.39:0/1007293429 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c0095f0 con 0x7f7c540ff410 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c53fff700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c5419c6d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400d8d0 tx=0x7f7c4400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c440098a0 con 0x7f7c540ff410 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7c44010460 con 0x7f7c540ff410 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c54196a50 con 0x7f7c540ff410 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c4400f5f0 con 0x7f7c540ff410 2024-02-19T19:01:28.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.914+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c54196fa0 con 0x7f7c540ff410 2024-02-19T19:01:28.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.915+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c541092b0 con 0x7f7c540ff410 2024-02-19T19:01:28.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.916+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7c44009a00 con 0x7f7c540ff410 2024-02-19T19:01:28.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.916+0000 7f7c517fa700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c4006c4e0 0x7f7c4006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:28.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.916+0000 7f7c537fe700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c4006c4e0 0x7f7c4006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:28.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.916+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7c4408b3d0 con 0x7f7c540ff410 2024-02-19T19:01:28.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.917+0000 7f7c537fe700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c4006c4e0 0x7f7c4006e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c54197f60 tx=0x7f7c3c01a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:28.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:28.921+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7c44059980 con 0x7f7c540ff410 2024-02-19T19:01:29.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.124+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7c54197da0 con 0x7f7c540ff410 2024-02-19T19:01:29.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.124+0000 7f7c517fa700 1 -- 172.21.15.39:0/1007293429 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c44059510 con 0x7f7c540ff410 2024-02-19T19:01:29.143 INFO:teuthology.orchestra.run.smithi039.stdout:81604378652 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c4006c4e0 msgr2=0x7f7c4006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c4006c4e0 0x7f7c4006e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c54197f60 tx=0x7f7c3c01a040 comp rx=0 tx=0).stop 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 msgr2=0x7f7c5419c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c5419c6d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400d8d0 tx=0x7f7c4400dbe0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 shutdown_connections 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f7c4006c4e0 0x7f7c4006e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c540ff410 0x7f7c5419c6d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 --2- 172.21.15.39:0/1007293429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c54106db0 0x7f7c5419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 >> 172.21.15.39:0/1007293429 conn(0x7f7c54074e30 msgr2=0x7f7c540fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:29.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 shutdown_connections 2024-02-19T19:01:29.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.126+0000 7f7c5a46d700 1 -- 172.21.15.39:0/1007293429 wait complete. 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2541879769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 msgr2=0x7ffbd8105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2541879769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd8105890 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009910 tx=0x7ffbcc009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2541879769 shutdown_connections 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2541879769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd8105890 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2541879769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8100e90 0x7ffbd8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2541879769 >> 172.21.15.39:0/2541879769 conn(0x7ffbd80fc740 msgr2=0x7ffbd80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2541879769 shutdown_connections 2024-02-19T19:01:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.135+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2541879769 wait complete. 2024-02-19T19:01:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbddbb5700 1 Processor -- start 2024-02-19T19:01:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbddbb5700 1 -- start start 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbddbb5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8100e90 0x7ffbd8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbddbb5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbddbb5700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd8073b10 con 0x7ffbd8101840 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbddbb5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd8073c50 con 0x7ffbd8100e90 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbd6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbd6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42738/0 (socket says 172.21.15.39:42738) 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.136+0000 7ffbd6ffd700 1 -- 172.21.15.39:0/2797418938 learned_addr learned my addr 172.21.15.39:0/2797418938 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd77fe700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8100e90 0x7ffbd8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd6ffd700 1 -- 172.21.15.39:0/2797418938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8100e90 msgr2=0x7ffbd8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd6ffd700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8100e90 0x7ffbd8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd6ffd700 1 -- 172.21.15.39:0/2797418938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc0095f0 con 0x7ffbd8101840 2024-02-19T19:01:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd6ffd700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd80735d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0092e0 tx=0x7ffbcc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbcc01d070 con 0x7ffbd8101840 2024-02-19T19:01:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffbcc022470 con 0x7ffbd8101840 2024-02-19T19:01:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd8073ed0 con 0x7ffbd8101840 2024-02-19T19:01:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbcc00f690 con 0x7ffbd8101840 2024-02-19T19:01:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.137+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd81a2e50 con 0x7ffbd8101840 2024-02-19T19:01:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.138+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd81091e0 con 0x7ffbd8101840 2024-02-19T19:01:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.139+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ffbcc0225e0 con 0x7ffbd8101840 2024-02-19T19:01:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.140+0000 7ffbd4ff9700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ffbc006c4e0 0x7ffbc006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.140+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffbcc08cac0 con 0x7ffbd8101840 2024-02-19T19:01:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.140+0000 7ffbd77fe700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ffbc006c4e0 0x7ffbc006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.140+0000 7ffbd77fe700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ffbc006c4e0 0x7ffbc006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8006fd0 tx=0x7ffbc8008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.145+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffbcc057690 con 0x7ffbd8101840 2024-02-19T19:01:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.344+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffbd8074660 con 0x7ffbd8101840 2024-02-19T19:01:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.345+0000 7ffbd4ff9700 1 -- 172.21.15.39:0/2797418938 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ffbcc05acb0 con 0x7ffbd8101840 2024-02-19T19:01:29.363 INFO:teuthology.orchestra.run.smithi039.stdout:111669149718 2024-02-19T19:01:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ffbc006c4e0 msgr2=0x7ffbc006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ffbc006c4e0 0x7ffbc006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8006fd0 tx=0x7ffbc8008040 comp rx=0 tx=0).stop 2024-02-19T19:01:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 msgr2=0x7ffbd80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd80735d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0092e0 tx=0x7ffbcc00be20 comp rx=0 tx=0).stop 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 shutdown_connections 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ffbc006c4e0 0x7ffbc006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8100e90 0x7ffbd8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 --2- 172.21.15.39:0/2797418938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbd8101840 0x7ffbd80735d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 >> 172.21.15.39:0/2797418938 conn(0x7ffbd80fc740 msgr2=0x7ffbd80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 shutdown_connections 2024-02-19T19:01:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:29.347+0000 7ffbddbb5700 1 -- 172.21.15.39:0/2797418938 wait complete. 2024-02-19T19:01:29.966 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:29 smithi039 ceph-mon[28988]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:29.966 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:29 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1007293429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T19:01:29.966 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:29 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2797418938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T19:01:29.987 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-02-19T19:01:29.987 DEBUG:teuthology.parallel:result is None 2024-02-19T19:01:30.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:29 smithi086 ceph-mon[37629]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:30.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:29 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1007293429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T19:01:30.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:29 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2797418938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T19:01:30.391 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378652 for osd.2 2024-02-19T19:01:30.392 DEBUG:teuthology.parallel:result is None 2024-02-19T19:01:30.392 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-19T19:01:30.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T19:01:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:30.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:31.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:32.006 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:31 smithi039 ceph-mon[28988]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:32.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:31 smithi086 ceph-mon[37629]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 -- 172.21.15.39:0/1095707869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 msgr2=0x7fe4040689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 --2- 172.21.15.39:0/1095707869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe4040689e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009910 tx=0x7fe3f8009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 -- 172.21.15.39:0/1095707869 shutdown_connections 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 --2- 172.21.15.39:0/1095707869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe4040689e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 --2- 172.21.15.39:0/1095707869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe404106cd0 0x7fe4041070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 -- 172.21.15.39:0/1095707869 >> 172.21.15.39:0/1095707869 conn(0x7fe404075670 msgr2=0x7fe404075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 -- 172.21.15.39:0/1095707869 shutdown_connections 2024-02-19T19:01:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.205+0000 7fe40beb6700 1 -- 172.21.15.39:0/1095707869 wait complete. 2024-02-19T19:01:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe40beb6700 1 Processor -- start 2024-02-19T19:01:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe40beb6700 1 -- start start 2024-02-19T19:01:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe40beb6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe40419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe40beb6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe404106cd0 0x7fe40419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe40beb6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40419d260 con 0x7fe404068580 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe40beb6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe404196760 con 0x7fe404106cd0 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe40419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe40419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42756/0 (socket says 172.21.15.39:42756) 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 -- 172.21.15.39:0/2536825855 learned_addr learned my addr 172.21.15.39:0/2536825855 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409451700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe404106cd0 0x7fe40419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 -- 172.21.15.39:0/2536825855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe404106cd0 msgr2=0x7fe40419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe404106cd0 0x7fe40419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 -- 172.21.15.39:0/2536825855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f80095f0 con 0x7fe404068580 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.206+0000 7fe409c52700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe40419c690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400ba50 tx=0x7fe3f400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.207+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3f400c900 con 0x7fe404068580 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.207+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe3f4014440 con 0x7fe404068580 2024-02-19T19:01:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.207+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe404196a40 con 0x7fe404068580 2024-02-19T19:01:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.207+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3f40075a0 con 0x7fe404068580 2024-02-19T19:01:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.207+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe404196f90 con 0x7fe404068580 2024-02-19T19:01:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.208+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4041091f0 con 0x7fe404068580 2024-02-19T19:01:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.208+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fe3f40145b0 con 0x7fe404068580 2024-02-19T19:01:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.209+0000 7fe402ffd700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3ec06c4e0 0x7fe3ec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.209+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe3f4006e40 con 0x7fe404068580 2024-02-19T19:01:32.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.209+0000 7fe409451700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3ec06c4e0 0x7fe3ec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:32.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.209+0000 7fe409451700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3ec06c4e0 0x7fe3ec06e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe404197b80 tx=0x7fe3f801a040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.213+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe3f408ce60 con 0x7fe404068580 2024-02-19T19:01:32.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.406+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe404197d30 con 0x7fe3ec06c4e0 2024-02-19T19:01:32.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.408+0000 7fe402ffd700 1 -- 172.21.15.39:0/2536825855 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27955 (secure 0 0 0) 0x7fe404197d30 con 0x7fe3ec06c4e0 2024-02-19T19:01:32.427 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:01:32.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3ec06c4e0 msgr2=0x7fe3ec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:32.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3ec06c4e0 0x7fe3ec06e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe404197b80 tx=0x7fe3f801a040 comp rx=0 tx=0).stop 2024-02-19T19:01:32.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 msgr2=0x7fe40419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:32.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe40419c690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400ba50 tx=0x7fe3f400bd60 comp rx=0 tx=0).stop 2024-02-19T19:01:32.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 shutdown_connections 2024-02-19T19:01:32.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3ec06c4e0 0x7fe3ec06e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe404068580 0x7fe40419c690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 --2- 172.21.15.39:0/2536825855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe404106cd0 0x7fe40419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 >> 172.21.15.39:0/2536825855 conn(0x7fe404075670 msgr2=0x7fe4040fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:32.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 shutdown_connections 2024-02-19T19:01:32.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:32.410+0000 7fe40beb6700 1 -- 172.21.15.39:0/2536825855 wait complete. 2024-02-19T19:01:32.430 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-02-19T19:01:33.073 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-02-19T19:01:30.697357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165584,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305328,"statfs":{"total":575970213888,"available":575800655872,"internally_reserved":0,"allocated":3964928,"data_stored":2608497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001891"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-19T19:00:42.919157+0000","last_change":"2024-02-19T19:00:16.990709+0000","last_active":"2024-02-19T19:00:42.919157+0000","last_peered":"2024-02-19T19:00:42.919157+0000","last_clean":"2024-02-19T19:00:42.919157+0000","last_became_active":"2024-02-19T19:00:16.990384+0000","last_became_peered":"2024-02-19T19:00:16.990384+0000","last_unstale":"2024-02-19T19:00:42.919157+0000","last_undegraded":"2024-02-19T19:00:42.919157+0000","last_fullsized":"2024-02-19T19:00:42.919157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T18:59:26.986958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T18:59:26.986958+0000","last_clean_scrub_stamp":"2024-02-19T18:59:26.986958+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:13.402014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.502}]},{"osd":2,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.432}]},{"osd":3,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":14.877000000000001,"5min":14.877000000000001,"15min":14.877000000000001},"last":0.438},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42199999999999999}]},{"osd":1,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.375},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":14.914999999999999,"5min":14.914999999999999,"15min":14.914999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Feb 19 19:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27304,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717848,"statfs":{"total":95995035648,"available":95967076352,"internally_reserved":0,"allocated":360448,"data_stored":139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:00:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 19:00:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Feb 19 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.379},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.44}]},{"osd":4,"last update":"Mon Feb 19 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 19:01:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"min":{"1min":0.20899999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.32500000000000001,"15min":0.317},"min":{"1min":0.27500000000000002,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.247}]},{"osd":2,"last update":"Mon Feb 19 19:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 19 19:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.434},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:01:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.317,"15min":0.311},"min":{"1min":0.23999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55300000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.441}]},{"osd":2,"last update":"Mon Feb 19 19:00:28 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Feb 19 19:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53500000000000003}]},{"osd":4,"last update":"Mon Feb 19 19:01:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T19:01:33.074 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T19:01:33.284 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:33.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:33 smithi039 ceph-mon[28988]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T19:01:33.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:33 smithi039 ceph-mon[28988]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:34.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:33 smithi086 ceph-mon[37629]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T19:01:34.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:33 smithi086 ceph-mon[37629]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:34.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/3137701737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f4101bb0 msgr2=0x7fc0f4101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:34.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/3137701737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f4101bb0 0x7fc0f4101f90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009910 tx=0x7fc0e8009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:34.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/3137701737 shutdown_connections 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/3137701737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f40ff340 0x7fc0f40ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/3137701737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f4101bb0 0x7fc0f4101f90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/3137701737 >> 172.21.15.39:0/3137701737 conn(0x7fc0f4074e30 msgr2=0x7fc0f4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.834+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/3137701737 shutdown_connections 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/3137701737 wait complete. 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 Processor -- start 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 -- start start 2024-02-19T19:01:34.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 0x7fc0f419e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f4101bb0 0x7fc0f419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f419f490 con 0x7fc0f40ff340 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0fb0cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f4198940 con 0x7fc0f4101bb0 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0f8e69700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 0x7fc0f419e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0f8e69700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 0x7fc0f419e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42774/0 (socket says 172.21.15.39:42774) 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.835+0000 7fc0f8e69700 1 -- 172.21.15.39:0/1063435424 learned_addr learned my addr 172.21.15.39:0/1063435424 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f8e69700 1 -- 172.21.15.39:0/1063435424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f4101bb0 msgr2=0x7fc0f419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f3fff700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f4101bb0 0x7fc0f419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f8e69700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f4101bb0 0x7fc0f419ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f8e69700 1 -- 172.21.15.39:0/1063435424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e80095f0 con 0x7fc0f40ff340 2024-02-19T19:01:34.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f8e69700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 0x7fc0f419e8c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006010 tx=0x7fc0e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:34.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e801d070 con 0x7fc0f40ff340 2024-02-19T19:01:34.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f4198bc0 con 0x7fc0f40ff340 2024-02-19T19:01:34.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc0e800f460 con 0x7fc0f40ff340 2024-02-19T19:01:34.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e8021640 con 0x7fc0f40ff340 2024-02-19T19:01:34.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.836+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f41990b0 con 0x7fc0f40ff340 2024-02-19T19:01:34.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.837+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f404f2e0 con 0x7fc0f40ff340 2024-02-19T19:01:34.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.838+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc0e800fab0 con 0x7fc0f40ff340 2024-02-19T19:01:34.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.838+0000 7fc0f1ffb700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc0e406c4e0 0x7fc0e406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:34.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.838+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc0e808c750 con 0x7fc0f40ff340 2024-02-19T19:01:34.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.838+0000 7fc0f3fff700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc0e406c4e0 0x7fc0e406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:34.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.839+0000 7fc0f3fff700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc0e406c4e0 0x7fc0e406e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0006fd0 tx=0x7fc0e0008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:34.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:34.843+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc0e805ad80 con 0x7fc0f40ff340 2024-02-19T19:01:35.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.037+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0f41993e0 con 0x7fc0e406c4e0 2024-02-19T19:01:35.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.038+0000 7fc0f1ffb700 1 -- 172.21.15.39:0/1063435424 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28022 (secure 0 0 0) 0x7fc0f41993e0 con 0x7fc0e406c4e0 2024-02-19T19:01:35.057 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:01:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc0e406c4e0 msgr2=0x7fc0e406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc0e406c4e0 0x7fc0e406e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0006fd0 tx=0x7fc0e0008040 comp rx=0 tx=0).stop 2024-02-19T19:01:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 msgr2=0x7fc0f419e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 0x7fc0f419e8c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006010 tx=0x7fc0e800be20 comp rx=0 tx=0).stop 2024-02-19T19:01:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 shutdown_connections 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc0e406c4e0 0x7fc0e406e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f40ff340 0x7fc0f419e8c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 --2- 172.21.15.39:0/1063435424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f4101bb0 0x7fc0f419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.040+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 >> 172.21.15.39:0/1063435424 conn(0x7fc0f4074e30 msgr2=0x7fc0f40fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.041+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 shutdown_connections 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:35.041+0000 7fc0fb0cd700 1 -- 172.21.15.39:0/1063435424 wait complete. 2024-02-19T19:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-02-19T19:01:35.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:34 smithi039 ceph-mon[28988]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:35.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:34 smithi086 ceph-mon[37629]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:35.788 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-02-19T19:01:34.698171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165584,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305328,"statfs":{"total":575970213888,"available":575800655872,"internally_reserved":0,"allocated":3964928,"data_stored":2608497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002081"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-19T19:00:42.919157+0000","last_change":"2024-02-19T19:00:16.990709+0000","last_active":"2024-02-19T19:00:42.919157+0000","last_peered":"2024-02-19T19:00:42.919157+0000","last_clean":"2024-02-19T19:00:42.919157+0000","last_became_active":"2024-02-19T19:00:16.990384+0000","last_became_peered":"2024-02-19T19:00:16.990384+0000","last_unstale":"2024-02-19T19:00:42.919157+0000","last_undegraded":"2024-02-19T19:00:42.919157+0000","last_fullsized":"2024-02-19T19:00:42.919157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T18:59:26.986958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T18:59:26.986958+0000","last_clean_scrub_stamp":"2024-02-19T18:59:26.986958+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T00:12:13.402014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.57499999999999996}]},{"osd":1,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Feb 19 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":14.877000000000001,"5min":14.877000000000001,"15min":14.877000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.436},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":14.914999999999999,"5min":14.914999999999999,"15min":14.914999999999999},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Feb 19 19:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.246},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.29999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27304,"kb_used_data":352,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717848,"statfs":{"total":95995035648,"available":95967076352,"internally_reserved":0,"allocated":360448,"data_stored":139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:01:28 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33200000000000002,"15min":0.33100000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.34999999999999998,"15min":0.35399999999999998},"min":{"1min":0.247,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.45400000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Feb 19 19:01:28 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35999999999999999,"15min":0.36199999999999999},"min":{"1min":0.248,"5min":0.221,"15min":0.221},"max":{"1min":0.499,"5min":0.5,"15min":0.5},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.47299999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.48499999999999999}]},{"osd":3,"last update":"Mon Feb 19 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.81299999999999994}]},{"osd":4,"last update":"Mon Feb 19 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 19:01:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.312,"15min":0.307},"min":{"1min":0.20899999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.32500000000000001,"15min":0.317},"min":{"1min":0.27500000000000002,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Feb 19 19:01:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.35799999999999998,"15min":0.36199999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39300000000000002,"15min":0.40000000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Feb 19 19:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Feb 19 19:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 19:01:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.317,"15min":0.311},"min":{"1min":0.23999999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.314},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.55300000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.373},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.50700000000000001,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.38600000000000001,"15min":0.38700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51000000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Feb 19 19:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Feb 19 19:01:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T19:01:35.789 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-19T19:01:35.789 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-19T19:01:35.789 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-19T19:01:35.789 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph health --format=json 2024-02-19T19:01:35.999 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:36.116 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:35 smithi039 ceph-mon[28988]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T19:01:36.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:35 smithi086 ceph-mon[37629]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T19:01:37.121 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:36 smithi039 ceph-mon[28988]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:37.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:36 smithi086 ceph-mon[37629]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 -- 172.21.15.39:0/3906484135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 msgr2=0x7fa328105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/3906484135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa328105840 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa314009910 tx=0x7fa314009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 -- 172.21.15.39:0/3906484135 shutdown_connections 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/3906484135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa328105840 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/3906484135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3280ff5a0 0x7fa3280ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 -- 172.21.15.39:0/3906484135 >> 172.21.15.39:0/3906484135 conn(0x7fa328074e30 msgr2=0x7fa328075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 -- 172.21.15.39:0/3906484135 shutdown_connections 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.628+0000 7fa32ed5b700 1 -- 172.21.15.39:0/3906484135 wait complete. 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32ed5b700 1 Processor -- start 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32ed5b700 1 -- start start 2024-02-19T19:01:37.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32ed5b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3280ff5a0 0x7fa32819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32ed5b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa32819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32ed5b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32819f4e0 con 0x7fa3280fff50 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32ed5b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa328198990 con 0x7fa3280ff5a0 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa327fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa32819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa327fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa32819ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39684/0 (socket says 172.21.15.39:39684) 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa327fff700 1 -- 172.21.15.39:0/4124043618 learned_addr learned my addr 172.21.15.39:0/4124043618 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.629+0000 7fa32caf7700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3280ff5a0 0x7fa32819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa327fff700 1 -- 172.21.15.39:0/4124043618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3280ff5a0 msgr2=0x7fa32819e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa327fff700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3280ff5a0 0x7fa32819e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa327fff700 1 -- 172.21.15.39:0/4124043618 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140095f0 con 0x7fa3280fff50 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa327fff700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa32819ee50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa314005f50 tx=0x7fa31400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa31401d070 con 0x7fa3280fff50 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa31400f460 con 0x7fa3280fff50 2024-02-19T19:01:37.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa328198c10 con 0x7fa3280fff50 2024-02-19T19:01:37.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa314017630 con 0x7fa3280fff50 2024-02-19T19:01:37.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.630+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa328199100 con 0x7fa3280fff50 2024-02-19T19:01:37.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.631+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa328104fc0 con 0x7fa3280fff50 2024-02-19T19:01:37.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.632+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa31400f5d0 con 0x7fa3280fff50 2024-02-19T19:01:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.632+0000 7fa325ffb700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa31806c4e0 0x7fa31806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.632+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa31408caa0 con 0x7fa3280fff50 2024-02-19T19:01:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.632+0000 7fa32caf7700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa31806c4e0 0x7fa31806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:37.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.633+0000 7fa32caf7700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa31806c4e0 0x7fa31806e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009a10 tx=0x7fa31c009190 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:37.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.637+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa31405b050 con 0x7fa3280fff50 2024-02-19T19:01:37.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.884+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa32804f2e0 con 0x7fa3280fff50 2024-02-19T19:01:37.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.884+0000 7fa325ffb700 1 -- 172.21.15.39:0/4124043618 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa31405abe0 con 0x7fa3280fff50 2024-02-19T19:01:37.903 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:01:37.903 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T19:01:37.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa31806c4e0 msgr2=0x7fa31806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:37.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa31806c4e0 0x7fa31806e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009a10 tx=0x7fa31c009190 comp rx=0 tx=0).stop 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 msgr2=0x7fa32819ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa32819ee50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa314005f50 tx=0x7fa31400bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 shutdown_connections 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa31806c4e0 0x7fa31806e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3280ff5a0 0x7fa32819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 --2- 172.21.15.39:0/4124043618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3280fff50 0x7fa32819ee50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 >> 172.21.15.39:0/4124043618 conn(0x7fa328074e30 msgr2=0x7fa3280fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 shutdown_connections 2024-02-19T19:01:37.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:37.887+0000 7fa32ed5b700 1 -- 172.21.15.39:0/4124043618 wait complete. 2024-02-19T19:01:38.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:37 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/4124043618' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T19:01:38.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:37 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/4124043618' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T19:01:38.518 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-19T19:01:38.519 INFO:tasks.cephadm:Setup complete, yielding 2024-02-19T19:01:38.519 INFO:teuthology.run_tasks:Running task print... 2024-02-19T19:01:38.527 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-02-19T19:01:38.527 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:01:38.566 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:01:38.567 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-19T19:01:38.778 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:39.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:38 smithi039 ceph-mon[28988]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:39.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:38 smithi086 ceph-mon[37629]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 -- 172.21.15.39:0/557136245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 msgr2=0x7fa96c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 --2- 172.21.15.39:0/557136245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c101270 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009910 tx=0x7fa95c009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 -- 172.21.15.39:0/557136245 shutdown_connections 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 --2- 172.21.15.39:0/557136245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c101270 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 --2- 172.21.15.39:0/557136245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c0ffc10 0x7fa96c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 -- 172.21.15.39:0/557136245 >> 172.21.15.39:0/557136245 conn(0x7fa96c0fb190 msgr2=0x7fa96c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.370+0000 7fa97446e700 1 -- 172.21.15.39:0/557136245 shutdown_connections 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 -- 172.21.15.39:0/557136245 wait complete. 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 Processor -- start 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 -- start start 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c0ffc10 0x7fa96c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c196fe0 con 0x7fa96c100e10 2024-02-19T19:01:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97446e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c197120 con 0x7fa96c0ffc10 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa971a09700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa971a09700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c1969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39712/0 (socket says 172.21.15.39:39712) 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa971a09700 1 -- 172.21.15.39:0/282886645 learned_addr learned my addr 172.21.15.39:0/282886645 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.371+0000 7fa97220a700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c0ffc10 0x7fa96c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa971a09700 1 -- 172.21.15.39:0/282886645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c0ffc10 msgr2=0x7fa96c196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa971a09700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c0ffc10 0x7fa96c196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa971a09700 1 -- 172.21.15.39:0/282886645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0095f0 con 0x7fa96c100e10 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa971a09700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c1969c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa95c006010 tx=0x7fa95c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa95c01d070 con 0x7fa96c100e10 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa95c00f460 con 0x7fa96c100e10 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa95c021640 con 0x7fa96c100e10 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96c19bb70 con 0x7fa96c100e10 2024-02-19T19:01:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.372+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96c19c060 con 0x7fa96c100e10 2024-02-19T19:01:40.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.373+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96c108700 con 0x7fa96c100e10 2024-02-19T19:01:40.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.374+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa95c00fab0 con 0x7fa96c100e10 2024-02-19T19:01:40.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.374+0000 7fa9637fe700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa95806c4e0 0x7fa95806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:40.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.374+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa95c08c620 con 0x7fa96c100e10 2024-02-19T19:01:40.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.374+0000 7fa97220a700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa95806c4e0 0x7fa95806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.374+0000 7fa97220a700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa95806c4e0 0x7fa95806e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa96c100c70 tx=0x7fa96800b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:40.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.379+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa95c05ac50 con 0x7fa96c100e10 2024-02-19T19:01:40.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.579+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fa96c19cab0 con 0x7fa96c100e10 2024-02-19T19:01:40.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.584+0000 7fa9637fe700 1 -- 172.21.15.39:0/282886645 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fa95c05a7e0 con 0x7fa96c100e10 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa95806c4e0 msgr2=0x7fa95806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa95806c4e0 0x7fa95806e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa96c100c70 tx=0x7fa96800b410 comp rx=0 tx=0).stop 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 msgr2=0x7fa96c1969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c1969c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa95c006010 tx=0x7fa95c00bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 shutdown_connections 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa95806c4e0 0x7fa95806e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c0ffc10 0x7fa96c196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 --2- 172.21.15.39:0/282886645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa96c100e10 0x7fa96c1969c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 >> 172.21.15.39:0/282886645 conn(0x7fa96c0fb190 msgr2=0x7fa96c068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:40.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 shutdown_connections 2024-02-19T19:01:40.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:40.586+0000 7fa97446e700 1 -- 172.21.15.39:0/282886645 wait complete. 2024-02-19T19:01:41.262 INFO:teuthology.run_tasks:Running task print... 2024-02-19T19:01:41.270 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-19T19:01:41.270 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:01:41.324 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:01:41.325 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph orch status' 2024-02-19T19:01:41.538 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:41.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:41 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/282886645' entity='client.admin' 2024-02-19T19:01:41.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:41.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:41 smithi039 ceph-mon[28988]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:41.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:41.925 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:01:41.925 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:01:42.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:41 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/282886645' entity='client.admin' 2024-02-19T19:01:42.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:42.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:41 smithi086 ceph-mon[37629]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:42.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:42.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:01:42.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 -- 172.21.15.39:0/3064932928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8104060 msgr2=0x7fabe8106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 --2- 172.21.15.39:0/3064932928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8104060 0x7fabe8106450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009910 tx=0x7fabdc009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 -- 172.21.15.39:0/3064932928 shutdown_connections 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 --2- 172.21.15.39:0/3064932928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8104060 0x7fabe8106450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 --2- 172.21.15.39:0/3064932928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8101730 0x7fabe8103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 -- 172.21.15.39:0/3064932928 >> 172.21.15.39:0/3064932928 conn(0x7fabe80fb120 msgr2=0x7fabe80fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 -- 172.21.15.39:0/3064932928 shutdown_connections 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.031+0000 7fabee156700 1 -- 172.21.15.39:0/3064932928 wait complete. 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabee156700 1 Processor -- start 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabee156700 1 -- start start 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabee156700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 0x7fabe8101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabee156700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8104060 0x7fabe80ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabee156700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80ffc20 con 0x7fabe8101730 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabee156700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80ffd60 con 0x7fabe8104060 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabe77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 0x7fabe8101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabe77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 0x7fabe8101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39720/0 (socket says 172.21.15.39:39720) 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabe77fe700 1 -- 172.21.15.39:0/1122627152 learned_addr learned my addr 172.21.15.39:0/1122627152 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.032+0000 7fabe6ffd700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8104060 0x7fabe80ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe77fe700 1 -- 172.21.15.39:0/1122627152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8104060 msgr2=0x7fabe80ff6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe77fe700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8104060 0x7fabe80ff6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe77fe700 1 -- 172.21.15.39:0/1122627152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc0095f0 con 0x7fabe8101730 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe6ffd700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8104060 0x7fabe80ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe77fe700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 0x7fabe8101090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fabd800b4b0 tx=0x7fabd800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd8011800 con 0x7fabe8101730 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fabd8011e10 con 0x7fabe8101730 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe8100040 con 0x7fabe8101730 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd800f2e0 con 0x7fabe8101730 2024-02-19T19:01:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.033+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe819e530 con 0x7fabe8101730 2024-02-19T19:01:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.034+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe8190690 con 0x7fabe8101730 2024-02-19T19:01:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.035+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fabd8010380 con 0x7fabe8101730 2024-02-19T19:01:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.035+0000 7fabe4ff9700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fabd006c4e0 0x7fabd006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.035+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fabd808b170 con 0x7fabe8101730 2024-02-19T19:01:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.035+0000 7fabe6ffd700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fabd006c4e0 0x7fabd006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.035+0000 7fabe6ffd700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fabd006c4e0 0x7fabd006e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0059d0 tx=0x7fabdc005920 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.039+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fabd8059720 con 0x7fabe8101730 2024-02-19T19:01:43.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.261+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabe80611d0 con 0x7fabd006c4e0 2024-02-19T19:01:43.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.262+0000 7fabe4ff9700 1 -- 172.21.15.39:0/1122627152 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fabe80611d0 con 0x7fabd006c4e0 2024-02-19T19:01:43.281 INFO:teuthology.orchestra.run.smithi039.stdout:Backend: cephadm 2024-02-19T19:01:43.281 INFO:teuthology.orchestra.run.smithi039.stdout:Available: Yes 2024-02-19T19:01:43.281 INFO:teuthology.orchestra.run.smithi039.stdout:Paused: No 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fabd006c4e0 msgr2=0x7fabd006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fabd006c4e0 0x7fabd006e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0059d0 tx=0x7fabdc005920 comp rx=0 tx=0).stop 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 msgr2=0x7fabe8101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 0x7fabe8101090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fabd800b4b0 tx=0x7fabd800b870 comp rx=0 tx=0).stop 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 shutdown_connections 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fabd006c4e0 0x7fabd006e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fabe8101730 0x7fabe8101090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 --2- 172.21.15.39:0/1122627152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabe8104060 0x7fabe80ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 >> 172.21.15.39:0/1122627152 conn(0x7fabe80fb120 msgr2=0x7fabe80fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:43.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 shutdown_connections 2024-02-19T19:01:43.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:43.264+0000 7fabee156700 1 -- 172.21.15.39:0/1122627152 wait complete. 2024-02-19T19:01:43.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-19T19:01:43.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:43 smithi039 ceph-mon[28988]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:44.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:43 smithi086 ceph-mon[37629]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:44.137 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:45.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:44 smithi086 ceph-mon[37629]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:45.203 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:44 smithi039 ceph-mon[28988]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:45.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 -- 172.21.15.39:0/1507375906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 msgr2=0x7f1e08102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:45.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1507375906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08102310 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009910 tx=0x7f1df8009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:45.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 -- 172.21.15.39:0/1507375906 shutdown_connections 2024-02-19T19:01:45.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1507375906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08102310 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:45.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1507375906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e08100cb0 0x7f1e081010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:45.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 -- 172.21.15.39:0/1507375906 >> 172.21.15.39:0/1507375906 conn(0x7f1e080fc230 msgr2=0x7f1e080fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:45.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 -- 172.21.15.39:0/1507375906 shutdown_connections 2024-02-19T19:01:45.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.698+0000 7f1e0e767700 1 -- 172.21.15.39:0/1507375906 wait complete. 2024-02-19T19:01:45.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e0e767700 1 Processor -- start 2024-02-19T19:01:45.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e0e767700 1 -- start start 2024-02-19T19:01:45.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e0e767700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e08100cb0 0x7f1e08074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:45.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e0e767700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e0e767700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080736c0 con 0x7f1e08101eb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e0e767700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08073800 con 0x7f1e08100cb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e077fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e077fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39750/0 (socket says 172.21.15.39:39750) 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e077fe700 1 -- 172.21.15.39:0/1740538894 learned_addr learned my addr 172.21.15.39:0/1740538894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e07fff700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e08100cb0 0x7f1e08074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e077fe700 1 -- 172.21.15.39:0/1740538894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e08100cb0 msgr2=0x7f1e08074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e077fe700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e08100cb0 0x7f1e08074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.699+0000 7f1e077fe700 1 -- 172.21.15.39:0/1740538894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80095f0 con 0x7f1e08101eb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.700+0000 7f1e077fe700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08073180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1df800b5c0 tx=0x7f1df800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.700+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df801d070 con 0x7f1e08101eb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.700+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e08073a80 con 0x7f1e08101eb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.700+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1df8022470 con 0x7f1e08101eb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.700+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df800f690 con 0x7f1e08101eb0 2024-02-19T19:01:45.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.700+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e08073f70 con 0x7f1e08101eb0 2024-02-19T19:01:45.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.701+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e081098b0 con 0x7f1e08101eb0 2024-02-19T19:01:45.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.701+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1df80225e0 con 0x7f1e08101eb0 2024-02-19T19:01:45.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.702+0000 7f1e057fa700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1df406c4e0 0x7f1df406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:45.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.702+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1df808cc80 con 0x7f1e08101eb0 2024-02-19T19:01:45.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.702+0000 7f1e07fff700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1df406c4e0 0x7f1df406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:45.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.702+0000 7f1e07fff700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1df406c4e0 0x7f1df406e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1e08101d10 tx=0x7f1df0009190 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:45.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.706+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1df805b230 con 0x7f1e08101eb0 2024-02-19T19:01:45.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.915+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e08106800 con 0x7f1df406c4e0 2024-02-19T19:01:45.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.924+0000 7f1e057fa700 1 -- 172.21.15.39:0/1740538894 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f1e08106800 con 0x7f1df406c4e0 2024-02-19T19:01:45.943 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:01:45.943 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (3m) 2m ago 5m 26.4M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:01:45.943 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (5m) 2m ago 5m 7059k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:01:45.943 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (4m) 64s ago 4m 6971k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (5m) 2m ago 5m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (4m) 64s ago 4m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (3m) 2m ago 5m 80.2M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:9283,8765,8443 running (6m) 2m ago 6m 485M - 18.2.1 94eadd990e0a f3c738bbd2c8 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (4m) 64s ago 4m 442M - 18.2.1 94eadd990e0a 1b78d388267f 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (6m) 2m ago 7m 45.7M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (4m) 64s ago 4m 40.3M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (5m) 2m ago 5m 14.3M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (4m) 64s ago 4m 16.0M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (3m) 2m ago 3m 43.4M 3946M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (2m) 2m ago 2m 40.7M 3946M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (2m) 2m ago 2m 39.7M 3946M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (2m) 64s ago 2m 41.5M 4970M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (96s) 64s ago 95s 39.2M 4970M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:01:45.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (69s) 64s ago 69s 14.2M 4970M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:01:45.945 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (3m) 2m ago 4m 38.8M - 2.43.0 a07b618ecd1d 54ef95da7db1 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1df406c4e0 msgr2=0x7f1df406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1df406c4e0 0x7f1df406e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1e08101d10 tx=0x7f1df0009190 comp rx=0 tx=0).stop 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 msgr2=0x7f1e08073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08073180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1df800b5c0 tx=0x7f1df800bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 shutdown_connections 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1df406c4e0 0x7f1df406e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e08100cb0 0x7f1e08074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 --2- 172.21.15.39:0/1740538894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e08101eb0 0x7f1e08073180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 >> 172.21.15.39:0/1740538894 conn(0x7f1e080fc230 msgr2=0x7f1e081050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 shutdown_connections 2024-02-19T19:01:45.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:45.926+0000 7f1e0e767700 1 -- 172.21.15.39:0/1740538894 wait complete. 2024-02-19T19:01:46.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:45 smithi086 ceph-mon[37629]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:46.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:46.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:45 smithi039 ceph-mon[28988]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:46.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:46.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-19T19:01:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:47.985 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:47 smithi039 ceph-mon[28988]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:47.986 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:47 smithi039 ceph-mon[28988]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:48.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:47 smithi086 ceph-mon[37629]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:47 smithi086 ceph-mon[37629]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:48.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- 172.21.15.39:0/2357039776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240100cb0 msgr2=0x7fc2401010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:48.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 --2- 172.21.15.39:0/2357039776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240100cb0 0x7fc2401010d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc234009910 tx=0x7fc234009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:48.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- 172.21.15.39:0/2357039776 shutdown_connections 2024-02-19T19:01:48.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 --2- 172.21.15.39:0/2357039776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc240101eb0 0x7fc240102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 --2- 172.21.15.39:0/2357039776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240100cb0 0x7fc2401010d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- 172.21.15.39:0/2357039776 >> 172.21.15.39:0/2357039776 conn(0x7fc2400fc230 msgr2=0x7fc2400fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- 172.21.15.39:0/2357039776 shutdown_connections 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- 172.21.15.39:0/2357039776 wait complete. 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 Processor -- start 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- start start 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc240100cb0 0x7fc2401964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 0x7fc240196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc240196fc0 con 0x7fc240101eb0 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.337+0000 7fc2471d9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc240197100 con 0x7fc240100cb0 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 0x7fc240196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 0x7fc240196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37428/0 (socket says 172.21.15.39:37428) 2024-02-19T19:01:48.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 -- 172.21.15.39:0/3849395444 learned_addr learned my addr 172.21.15.39:0/3849395444 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc244f75700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc240100cb0 0x7fc2401964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 -- 172.21.15.39:0/3849395444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc240100cb0 msgr2=0x7fc2401964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc240100cb0 0x7fc2401964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 -- 172.21.15.39:0/3849395444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340095f0 con 0x7fc240101eb0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23ffff700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 0x7fc240196a30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00c930 tx=0x7fc22c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22c007ad0 con 0x7fc240101eb0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc22c00ce80 con 0x7fc240101eb0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24019bbc0 con 0x7fc240101eb0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.338+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22c018630 con 0x7fc240101eb0 2024-02-19T19:01:48.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.339+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24019c0e0 con 0x7fc240101eb0 2024-02-19T19:01:48.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.339+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc240109810 con 0x7fc240101eb0 2024-02-19T19:01:48.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.340+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc22c007c30 con 0x7fc240101eb0 2024-02-19T19:01:48.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.340+0000 7fc23dffb700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc23006c4e0 0x7fc23006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:48.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.340+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc22c08b3a0 con 0x7fc240101eb0 2024-02-19T19:01:48.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.340+0000 7fc244f75700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc23006c4e0 0x7fc23006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:48.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.341+0000 7fc244f75700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc23006c4e0 0x7fc23006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc234006010 tx=0x7fc234005280 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:48.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.345+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc22c0599d0 con 0x7fc240101eb0 2024-02-19T19:01:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.548+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc240106800 con 0x7fc23006c4e0 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.554+0000 7fc23dffb700 1 -- 172.21.15.39:0/3849395444 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1290 (secure 0 0 0) 0x7fc240106800 con 0x7fc23006c4e0 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter 2/2 2m ago 6m * 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:crash 2/2 2m ago 6m * 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:mgr 2/2 2m ago 6m count:2 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:mon 2/2 2m ago 5m smithi039:172.21.15.39=smithi039;smithi086:172.21.15.86=smithi086;count:2 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:osd 6 2m ago - 2024-02-19T19:01:48.574 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-19T19:01:48.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc23006c4e0 msgr2=0x7fc23006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:48.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc23006c4e0 0x7fc23006e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc234006010 tx=0x7fc234005280 comp rx=0 tx=0).stop 2024-02-19T19:01:48.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 msgr2=0x7fc240196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:48.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 0x7fc240196a30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00c930 tx=0x7fc22c00cc40 comp rx=0 tx=0).stop 2024-02-19T19:01:48.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 shutdown_connections 2024-02-19T19:01:48.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc23006c4e0 0x7fc23006e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:48.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc240100cb0 0x7fc2401964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:48.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 --2- 172.21.15.39:0/3849395444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc240101eb0 0x7fc240196a30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:48.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 >> 172.21.15.39:0/3849395444 conn(0x7fc2400fc230 msgr2=0x7fc2401050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:48.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 shutdown_connections 2024-02-19T19:01:48.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:48.556+0000 7fc2471d9700 1 -- 172.21.15.39:0/3849395444 wait complete. 2024-02-19T19:01:49.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-19T19:01:49.494 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:49.945 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:49 smithi039 ceph-mon[28988]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:49.946 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:49 smithi039 ceph-mon[28988]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:50.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:49 smithi086 ceph-mon[37629]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:50.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:49 smithi086 ceph-mon[37629]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:51.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.095+0000 7f8a45a15700 1 -- 172.21.15.39:0/514125998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 msgr2=0x7f8a40103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.095+0000 7f8a45a15700 1 --2- 172.21.15.39:0/514125998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a40103b40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000b0a0 tx=0x7f8a3000b3b0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 -- 172.21.15.39:0/514125998 shutdown_connections 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 --2- 172.21.15.39:0/514125998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 0x7f8a40106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 --2- 172.21.15.39:0/514125998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a40103b40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 -- 172.21.15.39:0/514125998 >> 172.21.15.39:0/514125998 conn(0x7f8a400fb160 msgr2=0x7f8a400fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 -- 172.21.15.39:0/514125998 shutdown_connections 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 -- 172.21.15.39:0/514125998 wait complete. 2024-02-19T19:01:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 Processor -- start 2024-02-19T19:01:51.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 -- start start 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a401967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.096+0000 7f8a45a15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 0x7f8a40196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a45a15700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a40197350 con 0x7f8a40101750 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a45a15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a40197490 con 0x7f8a40104080 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a401967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a401967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37434/0 (socket says 172.21.15.39:37434) 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 -- 172.21.15.39:0/222194123 learned_addr learned my addr 172.21.15.39:0/222194123 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3e7fc700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 0x7f8a40196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 -- 172.21.15.39:0/222194123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 msgr2=0x7f8a40196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 0x7f8a40196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 -- 172.21.15.39:0/222194123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a30009d00 con 0x7f8a40101750 2024-02-19T19:01:51.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3e7fc700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 0x7f8a40196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a3effd700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a401967f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009fd0 tx=0x7f8a30009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a3000e040 con 0x7f8a40101750 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8a30003de0 con 0x7f8a40101750 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4019bee0 con 0x7f8a40101750 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a3001da10 con 0x7f8a40101750 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.097+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4019c350 con 0x7f8a40101750 2024-02-19T19:01:51.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.098+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a40190630 con 0x7f8a40101750 2024-02-19T19:01:51.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.099+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8a30004040 con 0x7f8a40101750 2024-02-19T19:01:51.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.099+0000 7f8a44a13700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8a2c06c4e0 0x7f8a2c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:51.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.099+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8a3008cea0 con 0x7f8a40101750 2024-02-19T19:01:51.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.099+0000 7f8a3e7fc700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8a2c06c4e0 0x7f8a2c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:51.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.100+0000 7f8a3e7fc700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8a2c06c4e0 0x7f8a2c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a28005950 tx=0x7f8a280058e0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:51.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.104+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8a3005b4d0 con 0x7f8a40101750 2024-02-19T19:01:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.310+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a400611d0 con 0x7f8a2c06c4e0 2024-02-19T19:01:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.312+0000 7f8a44a13700 1 -- 172.21.15.39:0/222194123 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f8a400611d0 con 0x7f8a2c06c4e0 2024-02-19T19:01:51.332 INFO:teuthology.orchestra.run.smithi039.stdout:HOST ADDR LABELS STATUS 2024-02-19T19:01:51.332 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039 172.21.15.39 2024-02-19T19:01:51.332 INFO:teuthology.orchestra.run.smithi039.stdout:smithi086 172.21.15.86 2024-02-19T19:01:51.332 INFO:teuthology.orchestra.run.smithi039.stdout:2 hosts in cluster 2024-02-19T19:01:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.314+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8a2c06c4e0 msgr2=0x7f8a2c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.314+0000 7f8a45a15700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8a2c06c4e0 0x7f8a2c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8a28005950 tx=0x7f8a280058e0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.314+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 msgr2=0x7f8a401967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.314+0000 7f8a45a15700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a401967f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009fd0 tx=0x7f8a30009790 comp rx=0 tx=0).stop 2024-02-19T19:01:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 shutdown_connections 2024-02-19T19:01:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f8a2c06c4e0 0x7f8a2c06e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a40101750 0x7f8a401967f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 --2- 172.21.15.39:0/222194123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a40104080 0x7f8a40196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 >> 172.21.15.39:0/222194123 conn(0x7f8a400fb160 msgr2=0x7f8a400fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 shutdown_connections 2024-02-19T19:01:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:51.315+0000 7f8a45a15700 1 -- 172.21.15.39:0/222194123 wait complete. 2024-02-19T19:01:51.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:51 smithi039 ceph-mon[28988]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:52.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-19T19:01:52.112 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:51 smithi086 ceph-mon[37629]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:53.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:52 smithi039 ceph-mon[28988]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:53.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:52 smithi086 ceph-mon[37629]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:53.885 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:53 smithi039 ceph-mon[28988]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:53.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.970+0000 7f562dc90700 1 -- 172.21.15.39:0/3202560134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 msgr2=0x7f5628103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:53.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.970+0000 7f562dc90700 1 --2- 172.21.15.39:0/3202560134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628103b90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5620009910 tx=0x7f5620009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 -- 172.21.15.39:0/3202560134 shutdown_connections 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 --2- 172.21.15.39:0/3202560134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56281040d0 0x7f56281064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 --2- 172.21.15.39:0/3202560134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628103b90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 -- 172.21.15.39:0/3202560134 >> 172.21.15.39:0/3202560134 conn(0x7f56280fb190 msgr2=0x7f56280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 -- 172.21.15.39:0/3202560134 shutdown_connections 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 -- 172.21.15.39:0/3202560134 wait complete. 2024-02-19T19:01:53.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 Processor -- start 2024-02-19T19:01:53.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 -- start start 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.971+0000 7f562dc90700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f562dc90700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56281040d0 0x7f56281969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f562dc90700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628196fc0 con 0x7f56281017a0 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f562dc90700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5628197100 con 0x7f56281040d0 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37456/0 (socket says 172.21.15.39:37456) 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 -- 172.21.15.39:0/3459169957 learned_addr learned my addr 172.21.15.39:0/3459169957 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f5626ffd700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56281040d0 0x7f56281969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 -- 172.21.15.39:0/3459169957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56281040d0 msgr2=0x7f56281969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56281040d0 0x7f56281969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 -- 172.21.15.39:0/3459169957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200095f0 con 0x7f56281017a0 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f56277fe700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628196460 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5620000c00 tx=0x7f562000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:53.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f562001d070 con 0x7f56281017a0 2024-02-19T19:01:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5620005dc0 con 0x7f56281017a0 2024-02-19T19:01:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562819bb50 con 0x7f56281017a0 2024-02-19T19:01:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56200226a0 con 0x7f56281017a0 2024-02-19T19:01:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.972+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562819c040 con 0x7f56281017a0 2024-02-19T19:01:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.973+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5628190610 con 0x7f56281017a0 2024-02-19T19:01:53.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.974+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f56200038f0 con 0x7f56281017a0 2024-02-19T19:01:53.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.974+0000 7f5624ff9700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f561406c4e0 0x7f561406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:53.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.974+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f562008c920 con 0x7f56281017a0 2024-02-19T19:01:53.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.974+0000 7f5626ffd700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f561406c4e0 0x7f561406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:53.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.975+0000 7f5626ffd700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f561406c4e0 0x7f561406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5610009c60 tx=0x7f5610009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:53.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:53.979+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f562005af50 con 0x7f56281017a0 2024-02-19T19:01:54.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.183+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f56280611d0 con 0x7f561406c4e0 2024-02-19T19:01:54.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.185+0000 7f5624ff9700 1 -- 172.21.15.39:0/3459169957 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f56280611d0 con 0x7f561406c4e0 2024-02-19T19:01:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-19T19:01:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102QX4P0KGN 3726G No 2m ago Has a FileSystem, LVM detected 2024-02-19T19:01:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:smithi086 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UL400BGN 372G No 69s ago Has a FileSystem, LVM detected 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f561406c4e0 msgr2=0x7f561406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f561406c4e0 0x7f561406e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5610009c60 tx=0x7f5610009260 comp rx=0 tx=0).stop 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 msgr2=0x7f5628196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628196460 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5620000c00 tx=0x7f562000f690 comp rx=0 tx=0).stop 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 shutdown_connections 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f561406c4e0 0x7f561406e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56281017a0 0x7f5628196460 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 --2- 172.21.15.39:0/3459169957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56281040d0 0x7f56281969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:54.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 >> 172.21.15.39:0/3459169957 conn(0x7f56280fb190 msgr2=0x7f56280fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:54.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 shutdown_connections 2024-02-19T19:01:54.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:54.187+0000 7f562dc90700 1 -- 172.21.15.39:0/3459169957 wait complete. 2024-02-19T19:01:54.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:53 smithi086 ceph-mon[37629]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:54.864 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-19T19:01:54.874 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:01:54.898 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:01:54.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-19T19:01:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:01:55.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:54 smithi039 ceph-mon[28988]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:55.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:54 smithi039 ceph-mon[28988]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:55.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:54 smithi086 ceph-mon[37629]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:55.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:54 smithi086 ceph-mon[37629]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 -- 172.21.15.39:0/23234301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 msgr2=0x7f6cdc101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/23234301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc101220 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009910 tx=0x7f6ccc009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 -- 172.21.15.39:0/23234301 shutdown_connections 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/23234301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc101220 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/23234301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc0ffbc0 0x7f6cdc0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 -- 172.21.15.39:0/23234301 >> 172.21.15.39:0/23234301 conn(0x7f6cdc0fb160 msgr2=0x7f6cdc0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:56.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 -- 172.21.15.39:0/23234301 shutdown_connections 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.737+0000 7f6ce34d0700 1 -- 172.21.15.39:0/23234301 wait complete. 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce34d0700 1 Processor -- start 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce34d0700 1 -- start start 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce34d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc0ffbc0 0x7f6cdc196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce34d0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce34d0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc1973d0 con 0x7f6cdc100dc0 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce34d0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc197510 con 0x7f6cdc0ffbc0 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce0a6b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce0a6b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39456/0 (socket says 172.21.15.39:39456) 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce0a6b700 1 -- 172.21.15.39:0/1964103304 learned_addr learned my addr 172.21.15.39:0/1964103304 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.738+0000 7f6ce126c700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc0ffbc0 0x7f6cdc196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6ce0a6b700 1 -- 172.21.15.39:0/1964103304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc0ffbc0 msgr2=0x7f6cdc196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6ce0a6b700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc0ffbc0 0x7f6cdc196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6ce0a6b700 1 -- 172.21.15.39:0/1964103304 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0095f0 con 0x7f6cdc100dc0 2024-02-19T19:01:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6ce0a6b700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc196db0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00b5c0 tx=0x7f6ccc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ccc01d070 con 0x7f6cdc100dc0 2024-02-19T19:01:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ccc022470 con 0x7f6cdc100dc0 2024-02-19T19:01:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc19bf60 con 0x7f6cdc100dc0 2024-02-19T19:01:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ccc00f690 con 0x7f6cdc100dc0 2024-02-19T19:01:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.739+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc19c420 con 0x7f6cdc100dc0 2024-02-19T19:01:56.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.740+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc068af0 con 0x7f6cdc100dc0 2024-02-19T19:01:56.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.740+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6ccc0225e0 con 0x7f6cdc100dc0 2024-02-19T19:01:56.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.741+0000 7f6cd27fc700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6cc806c4e0 0x7f6cc806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:56.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.741+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6ccc08cc90 con 0x7f6cdc100dc0 2024-02-19T19:01:56.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.741+0000 7f6ce126c700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6cc806c4e0 0x7f6cc806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:56.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.741+0000 7f6ce126c700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6cc806c4e0 0x7f6cc806e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc100c20 tx=0x7f6cd8009190 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.745+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6ccc05b240 con 0x7f6cdc100dc0 2024-02-19T19:01:56.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:56.975+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cdc069560 con 0x7f6cc806c4e0 2024-02-19T19:01:58.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:57 smithi086 ceph-mon[37629]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:58.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:57 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-19T19:01:58.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:57 smithi039 ceph-mon[28988]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:58.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:57 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-19T19:01:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.822+0000 7f6cd27fc700 1 -- 172.21.15.39:0/1964103304 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6cdc069560 con 0x7f6cc806c4e0 2024-02-19T19:01:58.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6cc806c4e0 msgr2=0x7f6cc806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:58.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6cc806c4e0 0x7f6cc806e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc100c20 tx=0x7f6cd8009190 comp rx=0 tx=0).stop 2024-02-19T19:01:58.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 msgr2=0x7f6cdc196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:58.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc196db0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00b5c0 tx=0x7f6ccc00bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 shutdown_connections 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f6cc806c4e0 0x7f6cc806e9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc0ffbc0 0x7f6cdc196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 --2- 172.21.15.39:0/1964103304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cdc100dc0 0x7f6cdc196db0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.825+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 >> 172.21.15.39:0/1964103304 conn(0x7f6cdc0fb160 msgr2=0x7f6cdc068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.826+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 shutdown_connections 2024-02-19T19:01:58.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:01:58.826+0000 7f6ce34d0700 1 -- 172.21.15.39:0/1964103304 wait complete. 2024-02-19T19:01:59.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:58 smithi039 ceph-mon[28988]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:59.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:58 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-19T19:01:59.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:58 smithi039 ceph-mon[28988]: osdmap e41: 6 total, 6 up, 6 in 2024-02-19T19:01:59.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:58 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:01:59.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[28984]: 2024-02-19T19:01:58.802+0000 7fa1c2611700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:01:59.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:58 smithi086 ceph-mon[37629]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:59.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:58 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-19T19:01:59.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:58 smithi086 ceph-mon[37629]: osdmap e41: 6 total, 6 up, 6 in 2024-02-19T19:01:59.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:58 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:01:59.570 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-19T19:01:59.804 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: pgmap v178: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:01:59.804 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: osdmap e42: 6 total, 6 up, 6 in 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: osdmap e43: 6 total, 6 up, 6 in 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: fsmap cephfs:0 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: Saving service mds.cephfs spec with placement count:4 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:01:59.805 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:01:59.806 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:01:59.806 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:01:59.806 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:01:59 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: pgmap v178: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: osdmap e42: 6 total, 6 up, 6 in 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: osdmap e43: 6 total, 6 up, 6 in 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-19T19:02:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: fsmap cephfs:0 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: Saving service mds.cephfs spec with placement count:4 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:00.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:01:59 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:00.357 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:01.120 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:00 smithi039 ceph-mon[28988]: Deploying daemon mds.cephfs.smithi039.omsmsr on smithi039 2024-02-19T19:02:01.120 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:00 smithi039 ceph-mon[28988]: osdmap e44: 6 total, 6 up, 6 in 2024-02-19T19:02:01.120 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:01.120 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:01.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:00 smithi086 ceph-mon[37629]: Deploying daemon mds.cephfs.smithi039.omsmsr on smithi039 2024-02-19T19:02:01.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:00 smithi086 ceph-mon[37629]: osdmap e44: 6 total, 6 up, 6 in 2024-02-19T19:02:01.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:01.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:01.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:01 smithi039 ceph-mon[28988]: pgmap v182: 65 pgs: 5 creating+activating, 3 creating+peering, 9 active+clean, 48 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:01.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:01 smithi039 ceph-mon[28988]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-19T19:02:01.973 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:01 smithi039 ceph-mon[28988]: osdmap e45: 6 total, 6 up, 6 in 2024-02-19T19:02:02.190 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:01 smithi086 ceph-mon[37629]: pgmap v182: 65 pgs: 5 creating+activating, 3 creating+peering, 9 active+clean, 48 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:02.190 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:01 smithi086 ceph-mon[37629]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-19T19:02:02.191 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:01 smithi086 ceph-mon[37629]: osdmap e45: 6 total, 6 up, 6 in 2024-02-19T19:02:02.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 -- 172.21.15.39:0/1596693680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 msgr2=0x7f405c100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 --2- 172.21.15.39:0/1596693680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c100030 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4058009910 tx=0x7f4058009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:02.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 -- 172.21.15.39:0/1596693680 shutdown_connections 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 --2- 172.21.15.39:0/1596693680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f405c100e10 0x7f405c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 --2- 172.21.15.39:0/1596693680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c100030 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 -- 172.21.15.39:0/1596693680 >> 172.21.15.39:0/1596693680 conn(0x7f405c0fb190 msgr2=0x7f405c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 -- 172.21.15.39:0/1596693680 shutdown_connections 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 -- 172.21.15.39:0/1596693680 wait complete. 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.518+0000 7f4063b6f700 1 Processor -- start 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f4063b6f700 1 -- start start 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f4063b6f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f4063b6f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f405c100e10 0x7f405c19d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f4063b6f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c19d690 con 0x7f405c0ffc10 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f4063b6f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c19d7d0 con 0x7f405c100e10 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c19cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39470/0 (socket says 172.21.15.39:39470) 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 -- 172.21.15.39:0/2649857447 learned_addr learned my addr 172.21.15.39:0/2649857447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 -- 172.21.15.39:0/2649857447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f405c100e10 msgr2=0x7f405c19d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406110a700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f405c100e10 0x7f405c19d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f405c100e10 0x7f405c19d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 -- 172.21.15.39:0/2649857447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580095f0 con 0x7f405c0ffc10 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f406190b700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c19cb30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f405800bda0 tx=0x7f405800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.519+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405801d070 con 0x7f405c0ffc10 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.520+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4058004c50 con 0x7f405c0ffc10 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.520+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c1a2220 con 0x7f405c0ffc10 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.520+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405800f690 con 0x7f405c0ffc10 2024-02-19T19:02:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.520+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1a2710 con 0x7f405c0ffc10 2024-02-19T19:02:02.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.521+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c108780 con 0x7f405c0ffc10 2024-02-19T19:02:02.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.521+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f40580229b0 con 0x7f405c0ffc10 2024-02-19T19:02:02.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.522+0000 7f4052ffd700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f404806c4e0 0x7f404806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.522+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f405808cf10 con 0x7f405c0ffc10 2024-02-19T19:02:02.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.522+0000 7f406110a700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f404806c4e0 0x7f404806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.522+0000 7f406110a700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f404806c4e0 0x7f404806e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f404c006fd0 tx=0x7f404c008040 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.526+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f405805b230 con 0x7f405c0ffc10 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.772+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f405c1a30a0 con 0x7f405c0ffc10 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.773+0000 7f4052ffd700 1 -- 172.21.15.39:0/2649857447 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7f405805adc0 con 0x7f405c0ffc10 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout:e2 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:02:02.793 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 2 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:01:58.803605+0000 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 0 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:in 2024-02-19T19:02:02.794 INFO:teuthology.orchestra.run.smithi039.stdout:up {} 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data disabled 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 0 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:02.795 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:02.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f404806c4e0 msgr2=0x7f404806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f404806c4e0 0x7f404806e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f404c006fd0 tx=0x7f404c008040 comp rx=0 tx=0).stop 2024-02-19T19:02:02.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 msgr2=0x7f405c19cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c19cb30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f405800bda0 tx=0x7f405800be80 comp rx=0 tx=0).stop 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 shutdown_connections 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f404806c4e0 0x7f404806e9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f405c0ffc10 0x7f405c19cb30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 --2- 172.21.15.39:0/2649857447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f405c100e10 0x7f405c19d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 >> 172.21.15.39:0/2649857447 conn(0x7f405c0fb190 msgr2=0x7f405c068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 shutdown_connections 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:02.775+0000 7f4063b6f700 1 -- 172.21.15.39:0/2649857447 wait complete. 2024-02-19T19:02:02.809 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 2 2024-02-19T19:02:03.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:02 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2649857447' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:03.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:02 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2649857447' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:03.665 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:02:03.674 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:03.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-02-19T19:02:03.892 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: pgmap v184: 65 pgs: 5 creating+activating, 3 creating+peering, 54 active+clean, 3 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:03.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:03 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: pgmap v184: 65 pgs: 5 creating+activating, 3 creating+peering, 54 active+clean, 3 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:03 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:05.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: Deploying daemon mds.cephfs.smithi086.euvbgw on smithi086 2024-02-19T19:02:05.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.39:6826/2541159330,v1:172.21.15.39:6827/2541159330] up:boot 2024-02-19T19:02:05.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: daemon mds.cephfs.smithi039.omsmsr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-19T19:02:05.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:02:05.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T19:02:05.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: Cluster is now healthy 2024-02-19T19:02:05.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: fsmap cephfs:0 1 up:standby 2024-02-19T19:02:05.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:02:05.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:creating} 2024-02-19T19:02:05.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:04 smithi086 ceph-mon[37629]: daemon mds.cephfs.smithi039.omsmsr is now active in filesystem cephfs as rank 0 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: Deploying daemon mds.cephfs.smithi086.euvbgw on smithi086 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.39:6826/2541159330,v1:172.21.15.39:6827/2541159330] up:boot 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: daemon mds.cephfs.smithi039.omsmsr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: Cluster is now healthy 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: fsmap cephfs:0 1 up:standby 2024-02-19T19:02:05.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:02:05.102 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:creating} 2024-02-19T19:02:05.102 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:04 smithi039 ceph-mon[28988]: daemon mds.cephfs.smithi039.omsmsr is now active in filesystem cephfs as rank 0 2024-02-19T19:02:05.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 -- 172.21.15.39:0/284896936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81040d0 msgr2=0x7f14f81064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:05.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/284896936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81040d0 0x7f14f81064c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009910 tx=0x7f14e0009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:05.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 -- 172.21.15.39:0/284896936 shutdown_connections 2024-02-19T19:02:05.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/284896936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81040d0 0x7f14f81064c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/284896936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14f81017a0 0x7f14f8103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 -- 172.21.15.39:0/284896936 >> 172.21.15.39:0/284896936 conn(0x7f14f80fb190 msgr2=0x7f14f80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 -- 172.21.15.39:0/284896936 shutdown_connections 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 -- 172.21.15.39:0/284896936 wait complete. 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.407+0000 7f14fd8fe700 1 Processor -- start 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14fd8fe700 1 -- start start 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14fd8fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 0x7f14f8101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14fd8fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14f81040d0 0x7f14f80ff750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14fd8fe700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80ffc90 con 0x7f14f81017a0 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14fd8fe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80ffdd0 con 0x7f14f81040d0 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 0x7f14f8101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 0x7f14f8101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39498/0 (socket says 172.21.15.39:39498) 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 -- 172.21.15.39:0/1160034707 learned_addr learned my addr 172.21.15.39:0/1160034707 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:05.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f67fc700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14f81040d0 0x7f14f80ff750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 -- 172.21.15.39:0/1160034707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14f81040d0 msgr2=0x7f14f80ff750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14f81040d0 0x7f14f80ff750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 -- 172.21.15.39:0/1160034707 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00095f0 con 0x7f14f81017a0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14f6ffd700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 0x7f14f8101100 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14f000ef10 tx=0x7f14f000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.408+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14f000ca50 con 0x7f14f81017a0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.409+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f8100050 con 0x7f14f81017a0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.409+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f14f000f040 con 0x7f14f81017a0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.409+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14f00146b0 con 0x7f14f81017a0 2024-02-19T19:02:05.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.409+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f81005a0 con 0x7f14f81017a0 2024-02-19T19:02:05.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.410+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f8190610 con 0x7f14f81017a0 2024-02-19T19:02:05.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.410+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f14f000cbb0 con 0x7f14f81017a0 2024-02-19T19:02:05.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.410+0000 7f14fc8fc700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f14e406c4e0 0x7f14e406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:05.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.410+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f14f008bc00 con 0x7f14f81017a0 2024-02-19T19:02:05.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.410+0000 7f14f67fc700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f14e406c4e0 0x7f14e406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:05.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.411+0000 7f14f67fc700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f14e406c4e0 0x7f14e406e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14e00059d0 tx=0x7f14e0005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:05.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.415+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f14f0091020 con 0x7f14f81017a0 2024-02-19T19:02:05.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.657+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f14f80619a0 con 0x7f14f81017a0 2024-02-19T19:02:05.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:05 smithi039 ceph-mon[28988]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:05.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:05 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.39:6826/2541159330,v1:172.21.15.39:6827/2541159330] up:active 2024-02-19T19:02:05.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:05 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 2024-02-19T19:02:05.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:05 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1160034707' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-19T19:02:05.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:05 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:05.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:05 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:05.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.971+0000 7f14fc8fc700 1 -- 172.21.15.39:0/1160034707 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f14f0059e70 con 0x7f14f81017a0 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f14e406c4e0 msgr2=0x7f14e406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f14e406c4e0 0x7f14e406e9a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14e00059d0 tx=0x7f14e0005940 comp rx=0 tx=0).stop 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 msgr2=0x7f14f8101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 0x7f14f8101100 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14f000ef10 tx=0x7f14f000c3a0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 shutdown_connections 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f14e406c4e0 0x7f14e406e9a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14f81017a0 0x7f14f8101100 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 --2- 172.21.15.39:0/1160034707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14f81040d0 0x7f14f80ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:05.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.973+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 >> 172.21.15.39:0/1160034707 conn(0x7f14f80fb190 msgr2=0x7f14f80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:05.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.974+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 shutdown_connections 2024-02-19T19:02:05.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:05.974+0000 7f14fd8fe700 1 -- 172.21.15.39:0/1160034707 wait complete. 2024-02-19T19:02:06.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:05 smithi086 ceph-mon[37629]: pgmap v185: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:06.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:05 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.39:6826/2541159330,v1:172.21.15.39:6827/2541159330] up:active 2024-02-19T19:02:06.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:05 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 2024-02-19T19:02:06.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:05 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1160034707' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-19T19:02:06.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:05 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:06.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:05 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:06.604 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:02:06.614 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:06.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-02-19T19:02:07.129 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:07.129 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: Deploying daemon mds.cephfs.smithi039.aygwnv on smithi039 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1160034707' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:boot 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:02:07.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:06 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby 2024-02-19T19:02:07.229 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: Deploying daemon mds.cephfs.smithi039.aygwnv on smithi039 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1160034707' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:boot 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:02:07.230 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:06 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby 2024-02-19T19:02:07.791 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:07.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:07 smithi039 ceph-mon[28988]: pgmap v186: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 389 B/s wr, 1 op/s 2024-02-19T19:02:08.072 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:07 smithi086 ceph-mon[37629]: pgmap v186: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 389 B/s wr, 1 op/s 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 -- 172.21.15.39:0/2017037586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8101750 msgr2=0x7f5cb8103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/2017037586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8101750 0x7f5cb8103b40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009910 tx=0x7f5ca8009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 -- 172.21.15.39:0/2017037586 shutdown_connections 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/2017037586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cb8104080 0x7f5cb8106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/2017037586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8101750 0x7f5cb8103b40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 -- 172.21.15.39:0/2017037586 >> 172.21.15.39:0/2017037586 conn(0x7f5cb80fb160 msgr2=0x7f5cb80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 -- 172.21.15.39:0/2017037586 shutdown_connections 2024-02-19T19:02:09.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 -- 172.21.15.39:0/2017037586 wait complete. 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 Processor -- start 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.457+0000 7f5cbdd08700 1 -- start start 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cbdd08700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cb8101750 0x7f5cb8196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cbdd08700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 0x7f5cb8196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cbdd08700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb8197380 con 0x7f5cb8104080 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cbdd08700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb81974c0 con 0x7f5cb8101750 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 0x7f5cb8196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 0x7f5cb8196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56692/0 (socket says 172.21.15.39:56692) 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 -- 172.21.15.39:0/1058642489 learned_addr learned my addr 172.21.15.39:0/1058642489 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb77fe700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cb8101750 0x7f5cb8196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 -- 172.21.15.39:0/1058642489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cb8101750 msgr2=0x7f5cb8196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cb8101750 0x7f5cb8196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:09.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 -- 172.21.15.39:0/1058642489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80095f0 con 0x7f5cb8104080 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb6ffd700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 0x7f5cb8196d60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00b4b0 tx=0x7f5cac00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cac011800 con 0x7f5cb8104080 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5cac011e10 con 0x7f5cb8104080 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb819bf70 con 0x7f5cb8104080 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.458+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cac00f2e0 con 0x7f5cb8104080 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.459+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb819c490 con 0x7f5cb8104080 2024-02-19T19:02:09.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.459+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb81906b0 con 0x7f5cb8104080 2024-02-19T19:02:09.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.460+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5cac010380 con 0x7f5cb8104080 2024-02-19T19:02:09.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.460+0000 7f5cb4ff9700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5ca006c4e0 0x7f5ca006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:09.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.460+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f5cac08b3c0 con 0x7f5cb8104080 2024-02-19T19:02:09.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.460+0000 7f5cb77fe700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5ca006c4e0 0x7f5ca006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:09.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.461+0000 7f5cb77fe700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5ca006c4e0 0x7f5ca006e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8006010 tx=0x7f5ca800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:09.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.465+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5cac059630 con 0x7f5cb8104080 2024-02-19T19:02:09.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.723+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f5cb8068af0 con 0x7f5cb8104080 2024-02-19T19:02:09.908 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:09 smithi039 ceph-mon[28988]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 4 op/s 2024-02-19T19:02:09.908 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:09 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1058642489' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-19T19:02:09.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.978+0000 7f5cb4ff9700 1 -- 172.21.15.39:0/1058642489 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v8) v1 ==== 121+0+0 (secure 0 0 0) 0x7f5cac0591c0 con 0x7f5cb8104080 2024-02-19T19:02:09.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5ca006c4e0 msgr2=0x7f5ca006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5ca006c4e0 0x7f5ca006e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8006010 tx=0x7f5ca800b540 comp rx=0 tx=0).stop 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 msgr2=0x7f5cb8196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 0x7f5cb8196d60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00b4b0 tx=0x7f5cac00b7c0 comp rx=0 tx=0).stop 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 shutdown_connections 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5ca006c4e0 0x7f5ca006e9a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cb8101750 0x7f5cb8196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 --2- 172.21.15.39:0/1058642489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cb8104080 0x7f5cb8196d60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.980+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 >> 172.21.15.39:0/1058642489 conn(0x7f5cb80fb160 msgr2=0x7f5cb80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.981+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 shutdown_connections 2024-02-19T19:02:10.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:09.981+0000 7f5cbdd08700 1 -- 172.21.15.39:0/1058642489 wait complete. 2024-02-19T19:02:10.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:09 smithi086 ceph-mon[37629]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 4 op/s 2024-02-19T19:02:10.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:09 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1058642489' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-19T19:02:10.591 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:02:10.602 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:10.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-19T19:02:10.814 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1058642489' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] up:boot 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 2 up:standby 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: Deploying daemon mds.cephfs.smithi086.idtugy on smithi086 2024-02-19T19:02:11.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-02-19T19:02:11.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:10 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1058642489' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] up:boot 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 2 up:standby 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: Deploying daemon mds.cephfs.smithi086.idtugy on smithi086 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-02-19T19:02:11.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:10 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:12.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1352518777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 msgr2=0x7fc5c0102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:12.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1352518777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0102310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009910 tx=0x7fc5b0009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:12.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1352518777 shutdown_connections 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1352518777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0102310 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1352518777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c0100cb0 0x7fc5c01010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1352518777 >> 172.21.15.39:0/1352518777 conn(0x7fc5c00fc230 msgr2=0x7fc5c00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1352518777 shutdown_connections 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1352518777 wait complete. 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.361+0000 7fc5c6c43700 1 Processor -- start 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c6c43700 1 -- start start 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c6c43700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c0100cb0 0x7fc5c01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c6c43700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c6c43700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0199210 con 0x7fc5c0101eb0 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c6c43700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0199350 con 0x7fc5c0100cb0 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56734/0 (socket says 172.21.15.39:56734) 2024-02-19T19:02:12.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 -- 172.21.15.39:0/1629310137 learned_addr learned my addr 172.21.15.39:0/1629310137 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c49df700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c0100cb0 0x7fc5c01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 -- 172.21.15.39:0/1629310137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c0100cb0 msgr2=0x7fc5c01986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c0100cb0 0x7fc5c01986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 -- 172.21.15.39:0/1629310137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00095f0 con 0x7fc5c0101eb0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bffff700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0198bf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5b001d070 con 0x7fc5c0101eb0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc5b000f460 con 0x7fc5c0101eb0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c019dda0 con 0x7fc5c0101eb0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.362+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5b0021640 con 0x7fc5c0101eb0 2024-02-19T19:02:12.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.363+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c019e290 con 0x7fc5c0101eb0 2024-02-19T19:02:12.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.364+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c0073810 con 0x7fc5c0101eb0 2024-02-19T19:02:12.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.364+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc5b000fab0 con 0x7fc5c0101eb0 2024-02-19T19:02:12.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.365+0000 7fc5bdffb700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc5ac06c4e0 0x7fc5ac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:12.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.365+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fc5b008ca70 con 0x7fc5c0101eb0 2024-02-19T19:02:12.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.365+0000 7fc5c49df700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc5ac06c4e0 0x7fc5ac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:12.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.365+0000 7fc5c49df700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc5ac06c4e0 0x7fc5ac06e9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009550 tx=0x7fc5b80091b0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:12.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.369+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc5b005ace0 con 0x7fc5c0101eb0 2024-02-19T19:02:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.614+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc5c0073a00 con 0x7fc5c0101eb0 2024-02-19T19:02:12.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.669+0000 7fc5bdffb700 1 -- 172.21.15.39:0/1629310137 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7fc5b005a870 con 0x7fc5c0101eb0 2024-02-19T19:02:12.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc5ac06c4e0 msgr2=0x7fc5ac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc5ac06c4e0 0x7fc5ac06e9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009550 tx=0x7fc5b80091b0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 msgr2=0x7fc5c0198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0198bf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b000bd70 comp rx=0 tx=0).stop 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 shutdown_connections 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fc5ac06c4e0 0x7fc5ac06e9a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5c0100cb0 0x7fc5c01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 --2- 172.21.15.39:0/1629310137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5c0101eb0 0x7fc5c0198bf0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 >> 172.21.15.39:0/1629310137 conn(0x7fc5c00fc230 msgr2=0x7fc5c00fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 shutdown_connections 2024-02-19T19:02:12.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:12.671+0000 7fc5c6c43700 1 -- 172.21.15.39:0/1629310137 wait complete. 2024-02-19T19:02:12.691 INFO:teuthology.orchestra.run.smithi039.stderr:inline data enabled 2024-02-19T19:02:12.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:12 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1629310137' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T19:02:13.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:12 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1629310137' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T19:02:13.440 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:02:13.451 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:13.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1629310137' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:boot 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 689 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:13 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:13.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-19T19:02:13.994 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1629310137' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:boot 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 689 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:13.995 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:13 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:14.791 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: Dropping low affinity active daemon mds.cephfs.smithi039.omsmsr in favor of higher affinity standby. 2024-02-19T19:02:14.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: Replacing daemon mds.cephfs.smithi039.omsmsr as rank 0 with standby daemon mds.cephfs.smithi086.euvbgw 2024-02-19T19:02:14.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:02:14.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: osdmap e46: 6 total, 6 up, 6 in 2024-02-19T19:02:14.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:standby-replay 2024-02-19T19:02:14.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] up:standby 2024-02-19T19:02:14.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:14 smithi039 ceph-mon[28988]: fsmap cephfs:1/1 {0=cephfs.smithi086.euvbgw=up:replay} 2 up:standby 2024-02-19T19:02:15.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: Dropping low affinity active daemon mds.cephfs.smithi039.omsmsr in favor of higher affinity standby. 2024-02-19T19:02:15.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: Replacing daemon mds.cephfs.smithi039.omsmsr as rank 0 with standby daemon mds.cephfs.smithi086.euvbgw 2024-02-19T19:02:15.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:02:15.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: osdmap e46: 6 total, 6 up, 6 in 2024-02-19T19:02:15.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:standby-replay 2024-02-19T19:02:15.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] up:standby 2024-02-19T19:02:15.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:14 smithi086 ceph-mon[37629]: fsmap cephfs:1/1 {0=cephfs.smithi086.euvbgw=up:replay} 2 up:standby 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:reconnect 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] up:boot 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: fsmap cephfs:1/1 {0=cephfs.smithi086.euvbgw=up:reconnect} 3 up:standby 2024-02-19T19:02:15.857 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:02:16.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-19T19:02:16.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:16.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:16.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:reconnect 2024-02-19T19:02:16.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] up:boot 2024-02-19T19:02:16.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: fsmap cephfs:1/1 {0=cephfs.smithi086.euvbgw=up:reconnect} 3 up:standby 2024-02-19T19:02:16.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.064+0000 7fec4ec05700 1 -- 172.21.15.39:0/2683198007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 msgr2=0x7fec481026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.064+0000 7fec4ec05700 1 --2- 172.21.15.39:0/2683198007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec481026b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009910 tx=0x7fec3c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.064+0000 7fec4ec05700 1 -- 172.21.15.39:0/2683198007 shutdown_connections 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.064+0000 7fec4ec05700 1 --2- 172.21.15.39:0/2683198007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec480fed10 0x7fec480ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.064+0000 7fec4ec05700 1 --2- 172.21.15.39:0/2683198007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec481026b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 -- 172.21.15.39:0/2683198007 >> 172.21.15.39:0/2683198007 conn(0x7fec480fa8b0 msgr2=0x7fec480fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 -- 172.21.15.39:0/2683198007 shutdown_connections 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 -- 172.21.15.39:0/2683198007 wait complete. 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 Processor -- start 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 -- start start 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec480fed10 0x7fec48196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec48196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec48197090 con 0x7fec48102290 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec4ec05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec481971d0 con 0x7fec480fed10 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec47fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec48196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec47fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec48196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56772/0 (socket says 172.21.15.39:56772) 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.065+0000 7fec47fff700 1 -- 172.21.15.39:0/1629486280 learned_addr learned my addr 172.21.15.39:0/1629486280 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec4c9a1700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec480fed10 0x7fec48196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec47fff700 1 -- 172.21.15.39:0/1629486280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec480fed10 msgr2=0x7fec48196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec47fff700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec480fed10 0x7fec48196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec47fff700 1 -- 172.21.15.39:0/1629486280 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c0095f0 con 0x7fec48102290 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec47fff700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec48196a70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fec3400d8d0 tx=0x7fec3400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec340098a0 con 0x7fec48102290 2024-02-19T19:02:16.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fec34010460 con 0x7fec48102290 2024-02-19T19:02:16.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4819bc80 con 0x7fec48102290 2024-02-19T19:02:16.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec3400f5f0 con 0x7fec48102290 2024-02-19T19:02:16.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.066+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4819c1d0 con 0x7fec48102290 2024-02-19T19:02:16.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.067+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec48109820 con 0x7fec48102290 2024-02-19T19:02:16.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.068+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fec34009a00 con 0x7fec48102290 2024-02-19T19:02:16.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.068+0000 7fec45ffb700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fec3806c4e0 0x7fec3806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:16.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.068+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fec3408b730 con 0x7fec48102290 2024-02-19T19:02:16.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.068+0000 7fec4c9a1700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fec3806c4e0 0x7fec3806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:16.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.068+0000 7fec4c9a1700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fec3806c4e0 0x7fec3806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fec3c006010 tx=0x7fec3c00b580 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:16.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.072+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fec34059940 con 0x7fec48102290 2024-02-19T19:02:16.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.323+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fec48109a10 con 0x7fec48102290 2024-02-19T19:02:16.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.324+0000 7fec45ffb700 1 -- 172.21.15.39:0/1629486280 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1828 (secure 0 0 0) 0x7fec34016740 con 0x7fec48102290 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:e13 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 13 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:15.974419+0000 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:rejoin seq 5 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{-1:14498} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:16.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fec3806c4e0 msgr2=0x7fec3806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:16.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fec3806c4e0 0x7fec3806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fec3c006010 tx=0x7fec3c00b580 comp rx=0 tx=0).stop 2024-02-19T19:02:16.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 msgr2=0x7fec48196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:16.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec48196a70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fec3400d8d0 tx=0x7fec3400dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 shutdown_connections 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fec3806c4e0 0x7fec3806e9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec480fed10 0x7fec48196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 --2- 172.21.15.39:0/1629486280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec48102290 0x7fec48196a70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 >> 172.21.15.39:0/1629486280 conn(0x7fec480fa8b0 msgr2=0x7fec48105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 shutdown_connections 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:16.326+0000 7fec4ec05700 1 -- 172.21.15.39:0/1629486280 wait complete. 2024-02-19T19:02:16.347 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 13 2024-02-19T19:02:16.830 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:16.830 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:16.830 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:16.830 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:rejoin 2024-02-19T19:02:16.831 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: fsmap cephfs:1/1 {0=cephfs.smithi086.euvbgw=up:rejoin} 3 up:standby 2024-02-19T19:02:16.831 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: daemon mds.cephfs.smithi086.euvbgw is now active in filesystem cephfs as rank 0 2024-02-19T19:02:16.831 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:16 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1629486280' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:17.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:17.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:17.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:17.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:rejoin 2024-02-19T19:02:17.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: fsmap cephfs:1/1 {0=cephfs.smithi086.euvbgw=up:rejoin} 3 up:standby 2024-02-19T19:02:17.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: daemon mds.cephfs.smithi086.euvbgw is now active in filesystem cephfs as rank 0 2024-02-19T19:02:17.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:16 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1629486280' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:17.057 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-19T19:02:17.453 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:17.913 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:17 smithi039 ceph-mon[28988]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-02-19T19:02:17.913 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:17 smithi039 ceph-mon[28988]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:02:17.913 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:17 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:active 2024-02-19T19:02:17.913 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:17 smithi039 ceph-mon[28988]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:standby 2024-02-19T19:02:17.913 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:17 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:02:17.913 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:17 smithi039 ceph-mon[28988]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:02:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:17 smithi086 ceph-mon[37629]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-02-19T19:02:18.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:17 smithi086 ceph-mon[37629]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:02:18.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:17 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] up:active 2024-02-19T19:02:18.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:17 smithi086 ceph-mon[37629]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:standby 2024-02-19T19:02:18.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:17 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:02:18.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:17 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:02:19.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:19 smithi039 ceph-mon[28988]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-19T19:02:20.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:19 smithi086 ceph-mon[37629]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-19T19:02:20.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 -- 172.21.15.39:0/1389942573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 msgr2=0x7efef01023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:20.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 --2- 172.21.15.39:0/1389942573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef01023a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efee4009910 tx=0x7efee4009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:20.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 -- 172.21.15.39:0/1389942573 shutdown_connections 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 --2- 172.21.15.39:0/1389942573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef01023a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 --2- 172.21.15.39:0/1389942573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efef00fef60 0x7efef00ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 -- 172.21.15.39:0/1389942573 >> 172.21.15.39:0/1389942573 conn(0x7efef00fa8c0 msgr2=0x7efef00fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 -- 172.21.15.39:0/1389942573 shutdown_connections 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 -- 172.21.15.39:0/1389942573 wait complete. 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 Processor -- start 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.362+0000 7efef7bdc700 1 -- start start 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef7bdc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efef00fef60 0x7efef01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef7bdc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef7bdc700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef0196fa0 con 0x7efef0101f40 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef7bdc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef01970e0 con 0x7efef00fef60 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5978700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efef00fef60 0x7efef01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53618/0 (socket says 172.21.15.39:53618) 2024-02-19T19:02:20.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 -- 172.21.15.39:0/318268403 learned_addr learned my addr 172.21.15.39:0/318268403 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:20.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 -- 172.21.15.39:0/318268403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efef00fef60 msgr2=0x7efef01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:20.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efef00fef60 0x7efef01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 -- 172.21.15.39:0/318268403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40095f0 con 0x7efef0101f40 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef5177700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef0196a10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efee400bd70 tx=0x7efee400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efee401d070 con 0x7efef0101f40 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef019bb40 con 0x7efef0101f40 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.363+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efee4005940 con 0x7efef0101f40 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.364+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efee4021430 con 0x7efef0101f40 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.364+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef019c000 con 0x7efef0101f40 2024-02-19T19:02:20.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.365+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef0109800 con 0x7efef0101f40 2024-02-19T19:02:20.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.365+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7efee400f860 con 0x7efef0101f40 2024-02-19T19:02:20.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.366+0000 7efeeeffd700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7efed806c4e0 0x7efed806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:20.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.366+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7efee408cc30 con 0x7efef0101f40 2024-02-19T19:02:20.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.366+0000 7efef5978700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7efed806c4e0 0x7efed806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:20.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.366+0000 7efef5978700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7efed806c4e0 0x7efed806e9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efef0068c10 tx=0x7efee0007a30 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:20.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.370+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efee4093080 con 0x7efef0101f40 2024-02-19T19:02:20.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.627+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef019c6d0 con 0x7efef0101f40 2024-02-19T19:02:20.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.628+0000 7efeeeffd700 1 -- 172.21.15.39:0/318268403 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4741 (secure 0 0 0) 0x7efee405ade0 con 0x7efef0101f40 2024-02-19T19:02:20.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7efed806c4e0 msgr2=0x7efed806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7efed806c4e0 0x7efed806e9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efef0068c10 tx=0x7efee0007a30 comp rx=0 tx=0).stop 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 msgr2=0x7efef0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef0196a10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efee400bd70 tx=0x7efee400bda0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 shutdown_connections 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7efed806c4e0 0x7efed806e9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efef00fef60 0x7efef01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 --2- 172.21.15.39:0/318268403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0101f40 0x7efef0196a10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 >> 172.21.15.39:0/318268403 conn(0x7efef00fa8c0 msgr2=0x7efef0105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 shutdown_connections 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:20.630+0000 7efef7bdc700 1 -- 172.21.15.39:0/318268403 wait complete. 2024-02-19T19:02:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:02:20.658 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:02:20.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:20 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/318268403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:02:21.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:20 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/318268403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:02:21.306 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-19T19:02:21.754 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:21.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:21 smithi039 ceph-mon[28988]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-19T19:02:22.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:21 smithi086 ceph-mon[37629]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-19T19:02:23.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 -- 172.21.15.39:0/3962038316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c100cb0 msgr2=0x7f691c1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 --2- 172.21.15.39:0/3962038316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c100cb0 0x7f691c1010d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6918009910 tx=0x7f6918009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 -- 172.21.15.39:0/3962038316 shutdown_connections 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 --2- 172.21.15.39:0/3962038316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c101eb0 0x7f691c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 --2- 172.21.15.39:0/3962038316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c100cb0 0x7f691c1010d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 -- 172.21.15.39:0/3962038316 >> 172.21.15.39:0/3962038316 conn(0x7f691c0fc230 msgr2=0x7f691c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 -- 172.21.15.39:0/3962038316 shutdown_connections 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.375+0000 7f69233df700 1 -- 172.21.15.39:0/3962038316 wait complete. 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.376+0000 7f69233df700 1 Processor -- start 2024-02-19T19:02:23.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.376+0000 7f69233df700 1 -- start start 2024-02-19T19:02:23.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.376+0000 7f69233df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 0x7f691c1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.376+0000 7f69233df700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 0x7f691c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.376+0000 7f69233df700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c197050 con 0x7f691c101eb0 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.376+0000 7f69233df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c197190 con 0x7f691c100cb0 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692097a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 0x7f691c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692117b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 0x7f691c1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692117b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 0x7f691c1964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:49038/0 (socket says 172.21.15.39:49038) 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692097a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 0x7f691c196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53626/0 (socket says 172.21.15.39:53626) 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692117b700 1 -- 172.21.15.39:0/93267158 learned_addr learned my addr 172.21.15.39:0/93267158 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692097a700 1 -- 172.21.15.39:0/93267158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 msgr2=0x7f691c1964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692097a700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 0x7f691c1964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692097a700 1 -- 172.21.15.39:0/93267158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69180095f0 con 0x7f691c101eb0 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692117b700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 0x7f691c1964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f692097a700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 0x7f691c196a30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f690c00d8d0 tx=0x7f690c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:23.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f690c00f860 con 0x7f691c101eb0 2024-02-19T19:02:23.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f690c00fe70 con 0x7f691c101eb0 2024-02-19T19:02:23.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c19bc40 con 0x7f691c101eb0 2024-02-19T19:02:23.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f690c00e5e0 con 0x7f691c101eb0 2024-02-19T19:02:23.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.377+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c19c190 con 0x7f691c101eb0 2024-02-19T19:02:23.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.378+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c109810 con 0x7f691c101eb0 2024-02-19T19:02:23.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.379+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f690c010460 con 0x7f691c101eb0 2024-02-19T19:02:23.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.379+0000 7f69127fc700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f690806c4e0 0x7f690806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:23.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.379+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f690c08b3c0 con 0x7f691c101eb0 2024-02-19T19:02:23.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.380+0000 7f692117b700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f690806c4e0 0x7f690806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:23.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.380+0000 7f692117b700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f690806c4e0 0x7f690806e9a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6918006010 tx=0x7f691800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:23.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.384+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f690c059650 con 0x7f691c101eb0 2024-02-19T19:02:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.698+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f691c109a00 con 0x7f691c101eb0 2024-02-19T19:02:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.698+0000 7f69127fc700 1 -- 172.21.15.39:0/93267158 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+83 (secure 0 0 0) 0x7f690c0591e0 con 0x7f691c101eb0 2024-02-19T19:02:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f690806c4e0 msgr2=0x7f690806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f690806c4e0 0x7f690806e9a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6918006010 tx=0x7f691800b540 comp rx=0 tx=0).stop 2024-02-19T19:02:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 msgr2=0x7f691c196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 0x7f691c196a30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f690c00d8d0 tx=0x7f690c00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 shutdown_connections 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f690806c4e0 0x7f690806e9a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f691c100cb0 0x7f691c1964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 --2- 172.21.15.39:0/93267158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691c101eb0 0x7f691c196a30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.700+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 >> 172.21.15.39:0/93267158 conn(0x7f691c0fc230 msgr2=0x7f691c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.701+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 shutdown_connections 2024-02-19T19:02:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:23.701+0000 7f69233df700 1 -- 172.21.15.39:0/93267158 wait complete. 2024-02-19T19:02:23.729 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:23.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:23 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/93267158' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-19T19:02:24.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:24.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:24.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:24.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:24.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:24.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:24.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:24.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:23 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/93267158' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-19T19:02:24.415 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-19T19:02:24.428 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 -- 172.21.15.39:0/822040179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c100180 msgr2=0x7f943c100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 --2- 172.21.15.39:0/822040179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c100180 0x7f943c100600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9424009880 tx=0x7f942402bce0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 -- 172.21.15.39:0/822040179 shutdown_connections 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 --2- 172.21.15.39:0/822040179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c100180 0x7f943c100600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 --2- 172.21.15.39:0/822040179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943c0fef90 0x7f943c0ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 -- 172.21.15.39:0/822040179 >> 172.21.15.39:0/822040179 conn(0x7f943c0fa500 msgr2=0x7f943c0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 -- 172.21.15.39:0/822040179 shutdown_connections 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.545+0000 7f9441399640 1 -- 172.21.15.39:0/822040179 wait complete. 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 Processor -- start 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 -- start start 2024-02-19T19:02:24.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 0x7f943c1948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943c100180 0x7f943c194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c1953c0 con 0x7f943c0fef90 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943c195530 con 0x7f943c100180 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 0x7f943c1948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 0x7f943c1948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53658/0 (socket says 172.21.15.39:53658) 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 -- 172.21.15.39:0/23690316 learned_addr learned my addr 172.21.15.39:0/23690316 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9433fff640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943c100180 0x7f943c194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 -- 172.21.15.39:0/23690316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943c100180 msgr2=0x7f943c194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943c100180 0x7f943c194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 -- 172.21.15.39:0/23690316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9424009560 con 0x7f943c0fef90 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f943affd640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 0x7f943c1948b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f942c00d8d0 tx=0x7f942c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f942c009a30 con 0x7f943c0fef90 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f942c009b90 con 0x7f943c0fef90 2024-02-19T19:02:24.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943c199fc0 con 0x7f943c0fef90 2024-02-19T19:02:24.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.546+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f942c00f680 con 0x7f943c0fef90 2024-02-19T19:02:24.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.547+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943c19a4c0 con 0x7f943c0fef90 2024-02-19T19:02:24.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.547+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943c107d20 con 0x7f943c0fef90 2024-02-19T19:02:24.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.548+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f942c009d00 con 0x7f943c0fef90 2024-02-19T19:02:24.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.548+0000 7f9438ff9640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f942806c450 0x7f942806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:24.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.548+0000 7f9433fff640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f942806c450 0x7f942806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:24.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.548+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f942c08b6e0 con 0x7f943c0fef90 2024-02-19T19:02:24.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.549+0000 7f9433fff640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f942806c450 0x7f942806e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f942402c7c0 tx=0x7f9424005fd0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:24.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.553+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f942c055e60 con 0x7f943c0fef90 2024-02-19T19:02:24.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.733+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f943c19a6e0 con 0x7f943c0fef90 2024-02-19T19:02:24.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.734+0000 7f9438ff9640 1 -- 172.21.15.39:0/23690316 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15116 (secure 0 0 0) 0x7f942c059480 con 0x7f943c0fef90 2024-02-19T19:02:24.753 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:24.753 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":46,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","created":"2024-02-19T18:54:43.947412+0000","modified":"2024-02-19T19:02:13.966563+0000","last_up_change":"2024-02-19T19:00:42.906219+0000","last_in_change":"2024-02-19T19:00:23.810529+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:59:26.986411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-19T19:01:57.224279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-19T19:01:58.037930+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91f217d9-d1a8-4658-b693-873f74d42408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6803","nonce":2663416145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6805","nonce":2663416145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6809","nonce":2663416145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2663416145},{"type":"v1","addr":"172.21.15.39:6807","nonce":2663416145}]},"public_addr":"172.21.15.39:6803/2663416145","cluster_addr":"172.21.15.39:6805/2663416145","heartbeat_back_addr":"172.21.15.39:6809/2663416145","heartbeat_front_addr":"172.21.15.39:6807/2663416145","state":["exists","up"]},{"osd":1,"uuid":"4b6143b3-8bce-4be7-a199-57711f652a9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6811","nonce":3708594483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6813","nonce":3708594483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6817","nonce":3708594483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3708594483},{"type":"v1","addr":"172.21.15.39:6815","nonce":3708594483}]},"public_addr":"172.21.15.39:6811/3708594483","cluster_addr":"172.21.15.39:6813/3708594483","heartbeat_back_addr":"172.21.15.39:6817/3708594483","heartbeat_front_addr":"172.21.15.39:6815/3708594483","state":["exists","up"]},{"osd":2,"uuid":"bcb75e1b-3f22-4e89-9bdd-ce8e833a3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6819","nonce":3928116015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6821","nonce":3928116015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6825","nonce":3928116015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":3928116015},{"type":"v1","addr":"172.21.15.39:6823","nonce":3928116015}]},"public_addr":"172.21.15.39:6819/3928116015","cluster_addr":"172.21.15.39:6821/3928116015","heartbeat_back_addr":"172.21.15.39:6825/3928116015","heartbeat_front_addr":"172.21.15.39:6823/3928116015","state":["exists","up"]},{"osd":3,"uuid":"8c3353e0-09f3-4582-af41-33ed9e1581c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6801","nonce":1886206958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6803","nonce":1886206958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6807","nonce":1886206958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1886206958},{"type":"v1","addr":"172.21.15.86:6805","nonce":1886206958}]},"public_addr":"172.21.15.86:6801/1886206958","cluster_addr":"172.21.15.86:6803/1886206958","heartbeat_back_addr":"172.21.15.86:6807/1886206958","heartbeat_front_addr":"172.21.15.86:6805/1886206958","state":["exists","up"]},{"osd":4,"uuid":"62e52e70-7bbd-4395-98b1-2bc8eabc2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6809","nonce":2218646412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6811","nonce":2218646412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6815","nonce":2218646412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2218646412},{"type":"v1","addr":"172.21.15.86:6813","nonce":2218646412}]},"public_addr":"172.21.15.86:6809/2218646412","cluster_addr":"172.21.15.86:6811/2218646412","heartbeat_back_addr":"172.21.15.86:6815/2218646412","heartbeat_front_addr":"172.21.15.86:6813/2218646412","state":["exists","up"]},{"osd":5,"uuid":"772c2485-d3a7-4ee6-9f6d-4f9535db5861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6817","nonce":1712053993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6819","nonce":1712053993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6823","nonce":1712053993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1712053993},{"type":"v1","addr":"172.21.15.86:6821","nonce":1712053993}]},"public_addr":"172.21.15.86:6817/1712053993","cluster_addr":"172.21.15.86:6819/1712053993","heartbeat_back_addr":"172.21.15.86:6823/1712053993","heartbeat_front_addr":"172.21.15.86:6821/1712053993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:24.714559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:54.443434+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:24.712567+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:49.208459+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:15.215844+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:41.317444+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6827/2541159330":"2024-02-20T19:02:13.966534+0000","172.21.15.39:0/2871115404":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/3613879910":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1614648654":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1444159856":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6800/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1261064923":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6801/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6801/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6801/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6800/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1749005598":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6826/2541159330":"2024-02-20T19:02:13.966534+0000","172.21.15.39:0/2313120383":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/2619661656":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/3933707157":"2024-02-20T18:57:00.645468+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f942806c450 msgr2=0x7f942806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f942806c450 0x7f942806e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f942402c7c0 tx=0x7f9424005fd0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 msgr2=0x7f943c1948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 0x7f943c1948b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f942c00d8d0 tx=0x7f942c00dd90 comp rx=0 tx=0).stop 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 shutdown_connections 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f942806c450 0x7f942806e910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943c100180 0x7f943c194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 --2- 172.21.15.39:0/23690316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f943c0fef90 0x7f943c1948b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 >> 172.21.15.39:0/23690316 conn(0x7f943c0fa500 msgr2=0x7f943c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:24.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 shutdown_connections 2024-02-19T19:02:24.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.736+0000 7f9441399640 1 -- 172.21.15.39:0/23690316 wait complete. 2024-02-19T19:02:24.766 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:59:26.986411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-19T19:01:57.224279+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-19T19:01:58.037930+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T19:02:24.767 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T19:02:24.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 -- 172.21.15.39:0/278679157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 msgr2=0x7f6820105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 --2- 172.21.15.39:0/278679157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f6820105840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f680c0098b0 tx=0x7f680c02bcd0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 -- 172.21.15.39:0/278679157 shutdown_connections 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 --2- 172.21.15.39:0/278679157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f6820105840 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 --2- 172.21.15.39:0/278679157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6820100b20 0x7f6820102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 -- 172.21.15.39:0/278679157 >> 172.21.15.39:0/278679157 conn(0x7f68200fa4f0 msgr2=0x7f68200fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 -- 172.21.15.39:0/278679157 shutdown_connections 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.853+0000 7f6826b39640 1 -- 172.21.15.39:0/278679157 wait complete. 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f6826b39640 1 Processor -- start 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f6826b39640 1 -- start start 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f6826b39640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6820100b20 0x7f6820100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f6826b39640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f68200fead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f6826b39640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200ff010 con 0x7f6820103450 2024-02-19T19:02:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f6826b39640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200ff180 con 0x7f6820100b20 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f68200fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f68200fead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53684/0 (socket says 172.21.15.39:53684) 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 -- 172.21.15.39:0/3512318468 learned_addr learned my addr 172.21.15.39:0/3512318468 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f68248d5640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6820100b20 0x7f6820100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 -- 172.21.15.39:0/3512318468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6820100b20 msgr2=0x7f6820100490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6820100b20 0x7f6820100490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 -- 172.21.15.39:0/3512318468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c009560 con 0x7f6820103450 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.854+0000 7f681ffff640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f68200fead0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.855+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f680c0044e0 con 0x7f6820103450 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.855+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f680c0054b0 con 0x7f6820103450 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.855+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68200ff400 con 0x7f6820103450 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.855+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f680c034440 con 0x7f6820103450 2024-02-19T19:02:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.855+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68200ff900 con 0x7f6820103450 2024-02-19T19:02:24.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.856+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6820108df0 con 0x7f6820103450 2024-02-19T19:02:24.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.856+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f680c003810 con 0x7f6820103450 2024-02-19T19:02:24.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.857+0000 7f681dffb640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f681006c450 0x7f681006e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:24.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.857+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f680c0adda0 con 0x7f6820103450 2024-02-19T19:02:24.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.857+0000 7f68248d5640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f681006c450 0x7f681006e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:24.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.857+0000 7f68248d5640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f681006c450 0x7f681006e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6814009b60 tx=0x7f6814005f90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:24.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:24.864+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f680c078520 con 0x7f6820103450 2024-02-19T19:02:25.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.053+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f682019da40 con 0x7f6820103450 2024-02-19T19:02:25.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.053+0000 7f681dffb640 1 -- 172.21.15.39:0/3512318468 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f680c07bb40 con 0x7f6820103450 2024-02-19T19:02:25.072 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-02-19T19:02:25.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f681006c450 msgr2=0x7f681006e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f681006c450 0x7f681006e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6814009b60 tx=0x7f6814005f90 comp rx=0 tx=0).stop 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 msgr2=0x7f68200fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f68200fead0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c004dc0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 shutdown_connections 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f681006c450 0x7f681006e910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6820103450 0x7f68200fead0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 --2- 172.21.15.39:0/3512318468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6820100b20 0x7f6820100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.055+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 >> 172.21.15.39:0/3512318468 conn(0x7f68200fa4f0 msgr2=0x7f68200fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.056+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 shutdown_connections 2024-02-19T19:02:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.056+0000 7f6826b39640 1 -- 172.21.15.39:0/3512318468 wait complete. 2024-02-19T19:02:25.086 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:24 smithi039 ceph-mon[28988]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-19T19:02:25.086 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:24 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/23690316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:02:25.087 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-19T19:02:25.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 -- 172.21.15.39:0/1972637215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 msgr2=0x7f75640730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 --2- 172.21.15.39:0/1972637215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f75640730e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f754c009880 tx=0x7f754c02bd70 comp rx=0 tx=0).stop 2024-02-19T19:02:25.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 -- 172.21.15.39:0/1972637215 shutdown_connections 2024-02-19T19:02:25.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 --2- 172.21.15.39:0/1972637215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f75640730e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 --2- 172.21.15.39:0/1972637215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7564074260 0x7f75640726b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 -- 172.21.15.39:0/1972637215 >> 172.21.15.39:0/1972637215 conn(0x7f75640fb610 msgr2=0x7f75640fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 -- 172.21.15.39:0/1972637215 shutdown_connections 2024-02-19T19:02:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.174+0000 7f75696d2640 1 -- 172.21.15.39:0/1972637215 wait complete. 2024-02-19T19:02:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f75696d2640 1 Processor -- start 2024-02-19T19:02:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f75696d2640 1 -- start start 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f75696d2640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f7564197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f75696d2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7564074260 0x7f7564198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f75696d2640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564198610 con 0x7f7564072c80 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f75696d2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564198780 con 0x7f7564074260 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f7564197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f7564197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53704/0 (socket says 172.21.15.39:53704) 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 -- 172.21.15.39:0/2319085200 learned_addr learned my addr 172.21.15.39:0/2319085200 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f755a7fc640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7564074260 0x7f7564198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 -- 172.21.15.39:0/2319085200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7564074260 msgr2=0x7f7564198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7564074260 0x7f7564198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 -- 172.21.15.39:0/2319085200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c009560 con 0x7f7564072c80 2024-02-19T19:02:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.175+0000 7f7562ffd640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f7564197b00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f755400e970 tx=0x7f755400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.176+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755400cc90 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.176+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f755400cdf0 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.176+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756419d210 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.176+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7554018810 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.176+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756419d740 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.177+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756410ae80 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.177+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f7554010c00 con 0x7f7564072c80 2024-02-19T19:02:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.177+0000 7f7560ff9640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f755006c450 0x7f755006e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.178+0000 7f755a7fc640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f755006c450 0x7f755006e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.178+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7554014070 con 0x7f7564072c80 2024-02-19T19:02:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.178+0000 7f755a7fc640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f755006c450 0x7f755006e910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f754c006010 tx=0x7f754c02c790 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.182+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f75540520e0 con 0x7f7564072c80 2024-02-19T19:02:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:24 smithi086 ceph-mon[37629]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-19T19:02:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:24 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/23690316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:02:25.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.362+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f756419dbe0 con 0x7f7564072c80 2024-02-19T19:02:25.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.363+0000 7f7560ff9640 1 -- 172.21.15.39:0/2319085200 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7554059fc0 con 0x7f7564072c80 2024-02-19T19:02:25.381 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-02-19T19:02:25.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f755006c450 msgr2=0x7f755006e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f755006c450 0x7f755006e910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f754c006010 tx=0x7f754c02c790 comp rx=0 tx=0).stop 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 msgr2=0x7f7564197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f7564197b00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f755400e970 tx=0x7f755400ee30 comp rx=0 tx=0).stop 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 shutdown_connections 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f755006c450 0x7f755006e910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7564074260 0x7f7564198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 --2- 172.21.15.39:0/2319085200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7564072c80 0x7f7564197b00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 >> 172.21.15.39:0/2319085200 conn(0x7f75640fb610 msgr2=0x7f7564105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 shutdown_connections 2024-02-19T19:02:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.365+0000 7f75696d2640 1 -- 172.21.15.39:0/2319085200 wait complete. 2024-02-19T19:02:25.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-19T19:02:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 -- 172.21.15.39:0/4193258442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 msgr2=0x7fcb60101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 --2- 172.21.15.39:0/4193258442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60101710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009880 tx=0x7fcb4802bce0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 -- 172.21.15.39:0/4193258442 shutdown_connections 2024-02-19T19:02:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 --2- 172.21.15.39:0/4193258442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60101710 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 --2- 172.21.15.39:0/4193258442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb601000a0 0x7fcb601004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 -- 172.21.15.39:0/4193258442 >> 172.21.15.39:0/4193258442 conn(0x7fcb600fb610 msgr2=0x7fcb600fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 -- 172.21.15.39:0/4193258442 shutdown_connections 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.480+0000 7fcb65577640 1 -- 172.21.15.39:0/4193258442 wait complete. 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb65577640 1 Processor -- start 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb65577640 1 -- start start 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb65577640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb601000a0 0x7fcb60195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb65577640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb65577640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb60196450 con 0x7fcb60101290 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb65577640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb601965c0 con 0x7fcb601000a0 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb5e7fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb5e7fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53730/0 (socket says 172.21.15.39:53730) 2024-02-19T19:02:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb5e7fc640 1 -- 172.21.15.39:0/2200672312 learned_addr learned my addr 172.21.15.39:0/2200672312 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.481+0000 7fcb5effd640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb601000a0 0x7fcb60195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb5e7fc640 1 -- 172.21.15.39:0/2200672312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb601000a0 msgr2=0x7fcb60195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb5e7fc640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb601000a0 0x7fcb60195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb5e7fc640 1 -- 172.21.15.39:0/2200672312 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009560 con 0x7fcb60101290 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb5e7fc640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60195e80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009230 tx=0x7fcb48004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb48004520 con 0x7fcb60101290 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcb480054b0 con 0x7fcb60101290 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6019aff0 con 0x7fcb60101290 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb48034670 con 0x7fcb60101290 2024-02-19T19:02:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.482+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6019b4c0 con 0x7fcb60101290 2024-02-19T19:02:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.483+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb60108c80 con 0x7fcb60101290 2024-02-19T19:02:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.483+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fcb480038b0 con 0x7fcb60101290 2024-02-19T19:02:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.484+0000 7fcb57fff640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcb4c06c450 0x7fcb4c06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.484+0000 7fcb5effd640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcb4c06c450 0x7fcb4c06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.484+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcb480adce0 con 0x7fcb60101290 2024-02-19T19:02:25.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.484+0000 7fcb5effd640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcb4c06c450 0x7fcb4c06e910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005fd0 tx=0x7fcb5800b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:25.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.488+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcb48078510 con 0x7fcb60101290 2024-02-19T19:02:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.665+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fcb60108e60 con 0x7fcb60101290 2024-02-19T19:02:25.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.665+0000 7fcb57fff640 1 -- 172.21.15.39:0/2200672312 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcb4807bb30 con 0x7fcb60101290 2024-02-19T19:02:25.684 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-02-19T19:02:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.667+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcb4c06c450 msgr2=0x7fcb4c06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcb4c06c450 0x7fcb4c06e910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb58005fd0 tx=0x7fcb5800b410 comp rx=0 tx=0).stop 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 msgr2=0x7fcb60195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60195e80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009230 tx=0x7fcb48004e30 comp rx=0 tx=0).stop 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 shutdown_connections 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcb4c06c450 0x7fcb4c06e910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb60101290 0x7fcb60195e80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 --2- 172.21.15.39:0/2200672312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb601000a0 0x7fcb60195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 >> 172.21.15.39:0/2200672312 conn(0x7fcb600fb610 msgr2=0x7fcb601044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 shutdown_connections 2024-02-19T19:02:25.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.668+0000 7fcb65577640 1 -- 172.21.15.39:0/2200672312 wait complete. 2024-02-19T19:02:25.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 -- 172.21.15.39:0/263942425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 msgr2=0x7fa7400ffd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 --2- 172.21.15.39:0/263942425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa7400ffd80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa73400a990 tx=0x7fa73402d010 comp rx=0 tx=0).stop 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 -- 172.21.15.39:0/263942425 shutdown_connections 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 --2- 172.21.15.39:0/263942425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa7400ffd80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 --2- 172.21.15.39:0/263942425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7400fe710 0x7fa7400feb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 -- 172.21.15.39:0/263942425 >> 172.21.15.39:0/263942425 conn(0x7fa7400f9c60 msgr2=0x7fa7400fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.782+0000 7fa7453eb640 1 -- 172.21.15.39:0/263942425 shutdown_connections 2024-02-19T19:02:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 -- 172.21.15.39:0/263942425 wait complete. 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 Processor -- start 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 -- start start 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7400fe710 0x7fa740193750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa740193c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740194260 con 0x7fa7400ff900 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa7453eb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401943d0 con 0x7fa7400fe710 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa73e7fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa740193c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa73e7fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa740193c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53754/0 (socket says 172.21.15.39:53754) 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.783+0000 7fa73e7fc640 1 -- 172.21.15.39:0/606912160 learned_addr learned my addr 172.21.15.39:0/606912160 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa73effd640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7400fe710 0x7fa740193750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa73e7fc640 1 -- 172.21.15.39:0/606912160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7400fe710 msgr2=0x7fa740193750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa73e7fc640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7400fe710 0x7fa740193750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa73e7fc640 1 -- 172.21.15.39:0/606912160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73400a670 con 0x7fa7400ff900 2024-02-19T19:02:25.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa73e7fc640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa740193c90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa73400a340 tx=0x7fa734005240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:25.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa734004420 con 0x7fa7400ff900 2024-02-19T19:02:25.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740198e00 con 0x7fa7400ff900 2024-02-19T19:02:25.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa734004b70 con 0x7fa7400ff900 2024-02-19T19:02:25.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa734003a50 con 0x7fa7400ff900 2024-02-19T19:02:25.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.784+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa740199330 con 0x7fa7400ff900 2024-02-19T19:02:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.785+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa740106a60 con 0x7fa7400ff900 2024-02-19T19:02:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.786+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa734035cf0 con 0x7fa7400ff900 2024-02-19T19:02:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.786+0000 7fa727fff640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa72806c450 0x7fa72806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.786+0000 7fa73effd640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa72806c450 0x7fa72806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.786+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa7340af7b0 con 0x7fa7400ff900 2024-02-19T19:02:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.787+0000 7fa73effd640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa72806c450 0x7fa72806e910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa730005fd0 tx=0x7fa73000b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:25.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:25.791+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa734079f30 con 0x7fa7400ff900 2024-02-19T19:02:26.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.015+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa740106c70 con 0x7fa7400ff900 2024-02-19T19:02:26.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.016+0000 7fa727fff640 1 -- 172.21.15.39:0/606912160 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fa73407d550 con 0x7fa7400ff900 2024-02-19T19:02:26.035 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:26.035 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14510,"name":"cephfs.smithi039.omsmsr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6827/438718767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":438718767},{"type":"v1","addr":"172.21.15.39:6827","nonce":438718767}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24279,"name":"cephfs.smithi086.idtugy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.86:6827/978975893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":978975893},{"type":"v1","addr":"172.21.15.86:6827","nonce":978975893}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T19:01:58.803530+0000","modified":"2024-02-19T19:02:16.986275+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14498":{"gid":14498,"name":"cephfs.smithi039.aygwnv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.39:6829/3608056816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":3608056816},{"type":"v1","addr":"172.21.15.39:6829","nonce":3608056816}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24267":{"gid":24267,"name":"cephfs.smithi086.euvbgw","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.86:6825/2684130153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2684130153},{"type":"v1","addr":"172.21.15.86:6825","nonce":2684130153}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-19T19:02:26.035 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa72806c450 msgr2=0x7fa72806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa72806c450 0x7fa72806e910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa730005fd0 tx=0x7fa73000b410 comp rx=0 tx=0).stop 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 msgr2=0x7fa740193c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa740193c90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa73400a340 tx=0x7fa734005240 comp rx=0 tx=0).stop 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 shutdown_connections 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa72806c450 0x7fa72806e910 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7400ff900 0x7fa740193c90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:26.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 --2- 172.21.15.39:0/606912160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7400fe710 0x7fa740193750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:26.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 >> 172.21.15.39:0/606912160 conn(0x7fa7400f9c60 msgr2=0x7fa740102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:26.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 shutdown_connections 2024-02-19T19:02:26.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:26.018+0000 7fa7453eb640 1 -- 172.21.15.39:0/606912160 wait complete. 2024-02-19T19:02:26.047 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 50} 2024-02-19T19:02:26.048 INFO:teuthology.run_tasks:Running task kclient... 2024-02-19T19:02:26.071 INFO:tasks.kclient:Mounting kernel clients... 2024-02-19T19:02:26.071 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-19T19:02:26.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:26.071 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-02-19T19:02:26.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:26.086 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-02-19T19:02:26.155 INFO:teuthology.orchestra.run.smithi039.stderr:Cannot find device "ceph-brx" 2024-02-19T19:02:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:02:26.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:26.157 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-02-19T19:02:26.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:26.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link delete ceph-brx 2024-02-19T19:02:26.243 INFO:teuthology.orchestra.run.smithi086.stderr:Cannot find device "ceph-brx" 2024-02-19T19:02:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:02:26.245 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-19T19:02:26.245 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T19:02:26.245 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:26.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-19T19:02:26.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:25 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3512318468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:02:26.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:25 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:26.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:25 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:26.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:25 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2319085200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:02:26.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:25 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2200672312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:02:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:25 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3512318468' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:02:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:25 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:25 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:25 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2319085200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:02:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:25 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2200672312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:02:26.551 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:27.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:26 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/606912160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:02:27.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:26 smithi086 ceph-mon[37629]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-19T19:02:27.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:26 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/606912160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:02:27.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:26 smithi039 ceph-mon[28988]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 -- 172.21.15.39:0/831215095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 msgr2=0x7f86840ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 --2- 172.21.15.39:0/831215095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f86840ff980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f867400b0a0 tx=0x7f867400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 -- 172.21.15.39:0/831215095 shutdown_connections 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 --2- 172.21.15.39:0/831215095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 0x7f8684105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 --2- 172.21.15.39:0/831215095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f86840ff980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 -- 172.21.15.39:0/831215095 >> 172.21.15.39:0/831215095 conn(0x7f8684074e30 msgr2=0x7f8684075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:28.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 -- 172.21.15.39:0/831215095 shutdown_connections 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.967+0000 7f868bbc6700 1 -- 172.21.15.39:0/831215095 wait complete. 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f868bbc6700 1 Processor -- start 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f868bbc6700 1 -- start start 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f868bbc6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f868419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f868bbc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 0x7f868419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f868bbc6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868419f450 con 0x7f86840ff5a0 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f868bbc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8684198900 con 0x7f86840fff50 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f8689962700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f868419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f8689962700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f868419e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57876/0 (socket says 172.21.15.39:57876) 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f8689962700 1 -- 172.21.15.39:0/1702759803 learned_addr learned my addr 172.21.15.39:0/1702759803 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f8689161700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 0x7f868419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f8689962700 1 -- 172.21.15.39:0/1702759803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 msgr2=0x7f868419edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.968+0000 7f8689962700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 0x7f868419edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:28.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f8689962700 1 -- 172.21.15.39:0/1702759803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674009d00 con 0x7f86840ff5a0 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f8689161700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 0x7f868419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f8689962700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f868419e880 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8674000f80 tx=0x7f867400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f867400e040 con 0x7f86840ff5a0 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8684198b80 con 0x7f86840ff5a0 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f86740044f0 con 0x7f86840ff5a0 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f867401d9f0 con 0x7f86840ff5a0 2024-02-19T19:02:28.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.969+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8684199070 con 0x7f86840ff5a0 2024-02-19T19:02:28.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.970+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8684104fc0 con 0x7f86840ff5a0 2024-02-19T19:02:28.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.971+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8674004660 con 0x7f86840ff5a0 2024-02-19T19:02:28.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.971+0000 7f8682ffd700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f866c06c4e0 0x7f866c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:28.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.971+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f867408d330 con 0x7f86840ff5a0 2024-02-19T19:02:28.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.971+0000 7f8689161700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f866c06c4e0 0x7f866c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:28.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.971+0000 7f8689161700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f866c06c4e0 0x7f866c06e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8684107740 tx=0x7f8678005f50 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:28.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:28.976+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f867405b540 con 0x7f86840ff5a0 2024-02-19T19:02:29.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.221+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f868404f2e0 con 0x7f86840ff5a0 2024-02-19T19:02:29.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.221+0000 7f8682ffd700 1 -- 172.21.15.39:0/1702759803 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f867405b0d0 con 0x7f86840ff5a0 2024-02-19T19:02:29.241 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-19T19:02:29.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f866c06c4e0 msgr2=0x7f866c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:29.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f866c06c4e0 0x7f866c06e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8684107740 tx=0x7f8678005f50 comp rx=0 tx=0).stop 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 msgr2=0x7f868419e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f868419e880 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8674000f80 tx=0x7f867400cdb0 comp rx=0 tx=0).stop 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 shutdown_connections 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f866c06c4e0 0x7f866c06e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86840ff5a0 0x7f868419e880 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.223+0000 7f868bbc6700 1 --2- 172.21.15.39:0/1702759803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86840fff50 0x7f868419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.224+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 >> 172.21.15.39:0/1702759803 conn(0x7f8684074e30 msgr2=0x7f86840fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.224+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 shutdown_connections 2024-02-19T19:02:29.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:29.224+0000 7f868bbc6700 1 -- 172.21.15.39:0/1702759803 wait complete. 2024-02-19T19:02:29.963 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T19:02:29.963 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T19:02:29.963 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-02-19T19:02:29.964 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T19:02:29.964 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:29.964 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T19:02:29.964 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T19:02:29.964 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T19:02:29.964 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T19:02:29.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:29.964 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:02:30.021 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42003sec preferred_lft 42003sec 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:02:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-02-19T19:02:30.023 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T19:02:30.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.023 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-02-19T19:02:30.023 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-02-19T19:02:30.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-02-19T19:02:30.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-02-19T19:02:30.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-02-19T19:02:30.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T19:02:30.024 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-02-19T19:02:30.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:29 smithi086 ceph-mon[37629]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-19T19:02:30.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:29 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1702759803' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:02:30.086 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:29 smithi039 ceph-mon[28988]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 7.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-19T19:02:30.087 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:29 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1702759803' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:02:30.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.203 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T19:02:30.272 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-02-19T19:02:30.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.274 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-02-19T19:02:30.346 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-02-19T19:02:30.346 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T19:02:30.346 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-19T19:02:30.347 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-19T19:02:30.347 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T19:02:30.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.347 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-02-19T19:02:30.347 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-02-19T19:02:30.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-19T19:02:30.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-19T19:02:30.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-19T19:02:30.347 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-02-19T19:02:30.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.628 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-02-19T19:02:30.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.643 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-02-19T19:02:30.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.697 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-02-19T19:02:30.697 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-02-19T19:02:30.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T19:02:30.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-19T19:02:30.697 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-02-19T19:02:30.780 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-19T19:02:30.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.780 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-02-19T19:02:30.780 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-02-19T19:02:30.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-19T19:02:30.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T19:02:30.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-19T19:02:30.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-19T19:02:30.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-19T19:02:30.781 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-02-19T19:02:30.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:30.923 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-02-19T19:02:30.923 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-02-19T19:02:30.923 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-02-19T19:02:30.923 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T19:02:30.923 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-02-19T19:02:30.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:02:30.964 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T19:02:31.018 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:02:31.018 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:31.072 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T19:02:31.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:31.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:02:31.168 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:02:31.168 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T19:02:31.168 INFO:teuthology.orchestra.run.smithi039.stdout:invalid new device string format 2024-02-19T19:02:31.277 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:02:31.277 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T19:02:31.277 INFO:teuthology.orchestra.run.smithi039.stdout:invalid new device string format 2024-02-19T19:02:31.277 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.39:3300,172.21.15.86:3300" 2024-02-19T19:02:31.277 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:invalid new device string format 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.39:3300,172.21.15.86:3300" 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.39:6789,172.21.15.86:6789" 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.39:6789,172.21.15.86:6789:/ 2024-02-19T19:02:31.283 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=1026b4de-cf58-11ee-95bb-87774f69a715" will pass to kernel 2024-02-19T19:02:31.284 INFO:teuthology.orchestra.run.smithi039.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-19T19:02:31.284 INFO:teuthology.orchestra.run.smithi039.stdout: You just mounted a file system that supports labels which does not 2024-02-19T19:02:31.284 INFO:teuthology.orchestra.run.smithi039.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T19:02:31.284 INFO:teuthology.orchestra.run.smithi039.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T19:02:31.284 INFO:teuthology.orchestra.run.smithi039.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T19:02:31.286 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T19:02:31.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:31.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:31.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:31.313 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:02:31.375 INFO:teuthology.orchestra.run.smithi039.stdout:client14548 2024-02-19T19:02:31.376 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T19:02:31.376 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:02:31.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:31.432 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:02:31.496 INFO:teuthology.orchestra.run.smithi039.stdout:client14548 2024-02-19T19:02:31.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:31.496 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:02:31.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:31 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:31.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:31.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:31 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:31.558 INFO:teuthology.orchestra.run.smithi039.stdout:client14548 2024-02-19T19:02:31.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:31.559 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/1026b4de-cf58-11ee-95bb-87774f69a715.client14548/status') 2024-02-19T19:02:31.629 INFO:teuthology.orchestra.run.smithi039.stdout:instance: client.14548 (3)192.168.0.1:0/2680228978 2024-02-19T19:02:31.629 INFO:teuthology.orchestra.run.smithi039.stdout:blocklisted: false 2024-02-19T19:02:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records in 2024-02-19T19:02:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records out 2024-02-19T19:02:31.630 INFO:teuthology.orchestra.run.smithi039.stderr:70 bytes copied, 2.0491e-05 s, 3.4 MB/s 2024-02-19T19:02:31.631 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-19T19:02:31.631 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T19:02:31.631 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:31.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-19T19:02:31.882 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 -- 172.21.15.39:0/3727374980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 msgr2=0x7f855010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 --2- 172.21.15.39:0/3727374980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f855010acd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8538009940 tx=0x7f8538009c50 comp rx=0 tx=0).stop 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 -- 172.21.15.39:0/3727374980 shutdown_connections 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 --2- 172.21.15.39:0/3727374980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f855010acd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 --2- 172.21.15.39:0/3727374980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 0x7f85501022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 -- 172.21.15.39:0/3727374980 >> 172.21.15.39:0/3727374980 conn(0x7f85500fb650 msgr2=0x7f85500fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 -- 172.21.15.39:0/3727374980 shutdown_connections 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.484+0000 7f8555595700 1 -- 172.21.15.39:0/3727374980 wait complete. 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f8555595700 1 Processor -- start 2024-02-19T19:02:33.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f8555595700 1 -- start start 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f8555595700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 0x7f85500ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f8555595700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f8550100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f8555595700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85501005a0 con 0x7f85501027e0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f8555595700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85501006e0 con 0x7f8550101ec0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.485+0000 7f854e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f8550100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f8550100060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57894/0 (socket says 172.21.15.39:57894) 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854e7fc700 1 -- 172.21.15.39:0/2943181528 learned_addr learned my addr 172.21.15.39:0/2943181528 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854effd700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 0x7f85500ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854e7fc700 1 -- 172.21.15.39:0/2943181528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 msgr2=0x7f85500ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854e7fc700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 0x7f85500ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854e7fc700 1 -- 172.21.15.39:0/2943181528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85380095f0 con 0x7f85501027e0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854effd700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 0x7f85500ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f854e7fc700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f8550100060 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8538005540 tx=0x7f8538005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f853801d070 con 0x7f85501027e0 2024-02-19T19:02:33.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8550100960 con 0x7f85501027e0 2024-02-19T19:02:33.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f853800f460 con 0x7f85501027e0 2024-02-19T19:02:33.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8538021640 con 0x7f85501027e0 2024-02-19T19:02:33.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.486+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85501a2d40 con 0x7f85501027e0 2024-02-19T19:02:33.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.487+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8550108360 con 0x7f85501027e0 2024-02-19T19:02:33.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.488+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f853800fab0 con 0x7f85501027e0 2024-02-19T19:02:33.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.488+0000 7f8547fff700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f853c06c4e0 0x7f853c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:33.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.488+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f853808cce0 con 0x7f85501027e0 2024-02-19T19:02:33.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.488+0000 7f854effd700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f853c06c4e0 0x7f853c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:33.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.489+0000 7f854effd700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f853c06c4e0 0x7f853c06e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8548005fd0 tx=0x7f8548005e30 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:33.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.493+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8538057520 con 0x7f85501027e0 2024-02-19T19:02:33.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.734+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f855004f2e0 con 0x7f85501027e0 2024-02-19T19:02:33.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.735+0000 7f8547fff700 1 -- 172.21.15.39:0/2943181528 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f853805ab40 con 0x7f85501027e0 2024-02-19T19:02:33.755 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f853c06c4e0 msgr2=0x7f853c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f853c06c4e0 0x7f853c06e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8548005fd0 tx=0x7f8548005e30 comp rx=0 tx=0).stop 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 msgr2=0x7f8550100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f8550100060 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8538005540 tx=0x7f8538005620 comp rx=0 tx=0).stop 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 shutdown_connections 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f853c06c4e0 0x7f853c06e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550101ec0 0x7f85500ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 --2- 172.21.15.39:0/2943181528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85501027e0 0x7f8550100060 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 >> 172.21.15.39:0/2943181528 conn(0x7f85500fb650 msgr2=0x7f85501054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 shutdown_connections 2024-02-19T19:02:33.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:33.737+0000 7f8555595700 1 -- 172.21.15.39:0/2943181528 wait complete. 2024-02-19T19:02:34.033 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:33 smithi086 ceph-mon[37629]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-19T19:02:34.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:33 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2943181528' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:02:34.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:33 smithi039 ceph-mon[28988]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-19T19:02:34.083 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:33 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2943181528' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:02:34.453 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T19:02:34.453 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T19:02:34.453 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi086.front.sepia.ceph.com 2024-02-19T19:02:34.453 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-19T19:02:34.454 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:34.454 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T19:02:34.454 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T19:02:34.454 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T19:02:34.454 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-19T19:02:34.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:34.454 DEBUG:teuthology.orchestra.run.smithi086:> ip addr 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 ::1/128 scope host 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f6 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f0 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f7 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:34.474 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f1 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:08 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:09 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout: inet 172.21.15.86/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft 42055sec preferred_lft 42055sec 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:02:34.475 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T19:02:34.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:34.475 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-02-19T19:02:34.476 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-02-19T19:02:34.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add name ceph-brx type bridge 2024-02-19T19:02:34.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip addr flush dev ceph-brx 2024-02-19T19:02:34.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set ceph-brx up 2024-02-19T19:02:34.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T19:02:34.476 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-02-19T19:02:34.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:34.651 DEBUG:teuthology.orchestra.run.smithi086:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T19:02:34.679 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-02-19T19:02:34.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:34.680 DEBUG:teuthology.orchestra.run.smithi086:> route 2024-02-19T19:02:34.749 INFO:teuthology.orchestra.run.smithi086.stdout:Kernel IP routing table 2024-02-19T19:02:34.749 INFO:teuthology.orchestra.run.smithi086.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T19:02:34.749 INFO:teuthology.orchestra.run.smithi086.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T19:02:34.749 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T19:02:34.749 INFO:teuthology.orchestra.run.smithi086.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T19:02:34.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:34.750 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-02-19T19:02:34.750 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-02-19T19:02:34.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T19:02:34.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T19:02:34.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T19:02:34.750 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-02-19T19:02:35.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.003 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-02-19T19:02:35.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.017 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list-id 2024-02-19T19:02:35.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.071 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-02-19T19:02:35.071 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-02-19T19:02:35.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T19:02:35.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-19T19:02:35.072 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-02-19T19:02:35.136 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:34 smithi086 ceph-mon[37629]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:02:35.155 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-19T19:02:35.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-19T19:02:35.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-19T19:02:35.156 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-02-19T19:02:35.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.296 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-02-19T19:02:35.296 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-02-19T19:02:35.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set brx.0 up 2024-02-19T19:02:35.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T19:02:35.296 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-02-19T19:02:35.337 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:02:35.337 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:35.392 INFO:teuthology.orchestra.run.smithi086.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-19T19:02:35.393 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:02:35.393 DEBUG:teuthology.orchestra.run.smithi086:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:35.447 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T19:02:35.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:02:35.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:34 smithi039 ceph-mon[28988]: pgmap v201: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:02:35.557 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:02:35.557 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T19:02:35.557 INFO:teuthology.orchestra.run.smithi086.stdout:invalid new device string format 2024-02-19T19:02:35.674 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:02:35.674 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T19:02:35.674 INFO:teuthology.orchestra.run.smithi086.stdout:invalid new device string format 2024-02-19T19:02:35.674 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: resolved to: "172.21.15.39:3300,172.21.15.86:3300" 2024-02-19T19:02:35.674 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:invalid new device string format 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: resolved to: "172.21.15.39:3300,172.21.15.86:3300" 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: resolved to: "172.21.15.39:6789,172.21.15.86:6789" 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.39:6789,172.21.15.86:6789:/ 2024-02-19T19:02:35.679 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=1026b4de-cf58-11ee-95bb-87774f69a715" will pass to kernel 2024-02-19T19:02:35.680 INFO:teuthology.orchestra.run.smithi086.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-19T19:02:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: You just mounted a file system that supports labels which does not 2024-02-19T19:02:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T19:02:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T19:02:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T19:02:35.682 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T19:02:35.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:35.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.708 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:02:35.764 INFO:teuthology.orchestra.run.smithi086.stdout:client14554 2024-02-19T19:02:35.765 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T19:02:35.765 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:02:35.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.821 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:02:35.877 INFO:teuthology.orchestra.run.smithi086.stdout:client14554 2024-02-19T19:02:35.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.877 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:02:35.933 INFO:teuthology.orchestra.run.smithi086.stdout:client14554 2024-02-19T19:02:35.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:02:35.933 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/1026b4de-cf58-11ee-95bb-87774f69a715.client14554/status') 2024-02-19T19:02:35.999 INFO:teuthology.orchestra.run.smithi086.stdout:instance: client.14554 (3)172.21.15.86:0/3079196984 2024-02-19T19:02:35.999 INFO:teuthology.orchestra.run.smithi086.stdout:blocklisted: false 2024-02-19T19:02:35.999 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2024-02-19T19:02:36.000 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2024-02-19T19:02:36.000 INFO:teuthology.orchestra.run.smithi086.stderr:71 bytes copied, 2.154e-05 s, 3.3 MB/s 2024-02-19T19:02:36.001 INFO:teuthology.run_tasks:Running task print... 2024-02-19T19:02:36.012 INFO:teuthology.task.print:**** done client 2024-02-19T19:02:36.012 INFO:teuthology.run_tasks:Running task parallel... 2024-02-19T19:02:36.053 INFO:teuthology.task.parallel:starting parallel... 2024-02-19T19:02:36.054 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-19T19:02:36.054 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T19:02:36.054 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:36.055 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-02-19T19:02:36.055 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-19T19:02:36.055 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-19T19:02:36.059 INFO:tasks.workunit:Pulling workunits from ref 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T19:02:36.059 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-19T19:02:36.059 INFO:tasks.workunit:timeout=3h 2024-02-19T19:02:36.059 INFO:tasks.workunit:cleanup=True 2024-02-19T19:02:36.059 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:36.075 INFO:teuthology.orchestra.run.smithi039.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5eh/94d Inode: 1 Links: 2 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-02-19 19:02:03.831292819 +0000 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-02-19 19:02:31.291986710 +0000 2024-02-19T19:02:36.076 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-02-19 19:02:03.831292819 +0000 2024-02-19T19:02:36.076 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-19T19:02:36.076 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-19T19:02:36.146 DEBUG:teuthology.orchestra.run.smithi086:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-02-19 19:02:36.122890393 +0000 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-02-19 19:02:36.122890393 +0000 2024-02-19T19:02:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-02-19 19:02:03.831292819 +0000 2024-02-19T19:02:36.163 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-19T19:02:36.163 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-19T19:02:36.232 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T19:02:36.232 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T19:02:36.248 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-19T19:02:36.283 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:36.287 INFO:tasks.workunit.client.1.smithi086.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-19T19:02:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 -- 172.21.15.39:0/2583063506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4104170 msgr2=0x7f11b4106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 --2- 172.21.15.39:0/2583063506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4104170 0x7f11b4106560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11a4009910 tx=0x7f11a4009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 -- 172.21.15.39:0/2583063506 shutdown_connections 2024-02-19T19:02:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 --2- 172.21.15.39:0/2583063506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4104170 0x7f11b4106560 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 --2- 172.21.15.39:0/2583063506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11b4101840 0x7f11b4103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 -- 172.21.15.39:0/2583063506 >> 172.21.15.39:0/2583063506 conn(0x7f11b40fb190 msgr2=0x7f11b40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 -- 172.21.15.39:0/2583063506 shutdown_connections 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 -- 172.21.15.39:0/2583063506 wait complete. 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 Processor -- start 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.898+0000 7f11bb426700 1 -- start start 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11bb426700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 0x7f11b4196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11bb426700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11b4104170 0x7f11b41969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11bb426700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4196fd0 con 0x7f11b4101840 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11bb426700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4197110 con 0x7f11b4104170 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 0x7f11b4196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 0x7f11b4196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51560/0 (socket says 172.21.15.39:51560) 2024-02-19T19:02:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 -- 172.21.15.39:0/3032515120 learned_addr learned my addr 172.21.15.39:0/3032515120 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 -- 172.21.15.39:0/3032515120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11b4104170 msgr2=0x7f11b41969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11b4104170 0x7f11b41969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 -- 172.21.15.39:0/3032515120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a40095f0 con 0x7f11b4101840 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11b91c2700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 0x7f11b4196470 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f11b000d8d0 tx=0x7f11b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11b0009940 con 0x7f11b4101840 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b419bbc0 con 0x7f11b4101840 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11b0010460 con 0x7f11b4101840 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11b000f5f0 con 0x7f11b4101840 2024-02-19T19:02:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.899+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b419c110 con 0x7f11b4101840 2024-02-19T19:02:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.900+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b4109810 con 0x7f11b4101840 2024-02-19T19:02:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.901+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f11b00105d0 con 0x7f11b4101840 2024-02-19T19:02:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.901+0000 7f11aa7fc700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f11a006c4e0 0x7f11a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.901+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f11b008b890 con 0x7f11b4101840 2024-02-19T19:02:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.901+0000 7f11b89c1700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f11a006c4e0 0x7f11a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.902+0000 7f11b89c1700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f11a006c4e0 0x7f11a006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f11a4006010 tx=0x7f11a400b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:37.906+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f11b0059aa0 con 0x7f11b4101840 2024-02-19T19:02:37.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:37 smithi039 ceph-mon[28988]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-02-19T19:02:38.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.102+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f11b419c870 con 0x7f11b4101840 2024-02-19T19:02:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.111+0000 7f11aa7fc700 1 -- 172.21.15.39:0/3032515120 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7f11b0059630 con 0x7f11b4101840 2024-02-19T19:02:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f11a006c4e0 msgr2=0x7f11a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f11a006c4e0 0x7f11a006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f11a4006010 tx=0x7f11a400b540 comp rx=0 tx=0).stop 2024-02-19T19:02:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 msgr2=0x7f11b4196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 0x7f11b4196470 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f11b000d8d0 tx=0x7f11b000dc90 comp rx=0 tx=0).stop 2024-02-19T19:02:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 shutdown_connections 2024-02-19T19:02:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f11a006c4e0 0x7f11a006e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11b4101840 0x7f11b4196470 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 --2- 172.21.15.39:0/3032515120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11b4104170 0x7f11b41969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 >> 172.21.15.39:0/3032515120 conn(0x7f11b40fb190 msgr2=0x7f11b40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 shutdown_connections 2024-02-19T19:02:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:38.113+0000 7f11bb426700 1 -- 172.21.15.39:0/3032515120 wait complete. 2024-02-19T19:02:38.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:37 smithi086 ceph-mon[37629]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 2 op/s 2024-02-19T19:02:38.853 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T19:02:38.854 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:38.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-19T19:02:39.108 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:39.459 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:39 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3032515120' entity='client.admin' 2024-02-19T19:02:39.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:39.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:39.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:39.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:39.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:39 smithi039 ceph-mon[28988]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T19:02:39.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:39 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3032515120' entity='client.admin' 2024-02-19T19:02:39.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:39.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:39.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:39.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:39.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:39 smithi086 ceph-mon[37629]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/567683981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8101eb0 msgr2=0x7f0ec8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/567683981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8102310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009910 tx=0x7f0eb0009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/567683981 shutdown_connections 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/567683981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8102310 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/567683981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8100cb0 0x7f0ec81010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/567683981 >> 172.21.15.39:0/567683981 conn(0x7f0ec80fc230 msgr2=0x7f0ec80fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/567683981 shutdown_connections 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/567683981 wait complete. 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.754+0000 7f0ecd3d7700 1 Processor -- start 2024-02-19T19:02:40.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ecd3d7700 1 -- start start 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ecd3d7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 0x7f0ec8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ecd3d7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ecd3d7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80736c0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ecd3d7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec8073800 con 0x7f0ec8101eb0 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 0x7f0ec8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 0x7f0ec8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51580/0 (socket says 172.21.15.39:51580) 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 -- 172.21.15.39:0/3892122770 learned_addr learned my addr 172.21.15.39:0/3892122770 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec67fc700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 -- 172.21.15.39:0/3892122770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8101eb0 msgr2=0x7f0ec8073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 -- 172.21.15.39:0/3892122770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00095f0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec67fc700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:02:40.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ec6ffd700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 0x7f0ec8074b30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0007e90 tx=0x7f0ec000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:40.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.755+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ec0017070 con 0x7f0ec8100cb0 2024-02-19T19:02:40.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.756+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0ec000f040 con 0x7f0ec8100cb0 2024-02-19T19:02:40.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.756+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec8073ae0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.756+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ec0004880 con 0x7f0ec8100cb0 2024-02-19T19:02:40.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.756+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec819e510 con 0x7f0ec8100cb0 2024-02-19T19:02:40.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.757+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec8109810 con 0x7f0ec8100cb0 2024-02-19T19:02:40.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.757+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f0ec00040b0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.758+0000 7f0ebffff700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0eb406c490 0x7f0eb406e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:40.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.758+0000 7f0ec67fc700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0eb406c490 0x7f0eb406e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:40.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.758+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0ec008b2d0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.758+0000 7f0ec67fc700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0eb406c490 0x7f0eb406e950 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000b5c0 tx=0x7f0eb0005e20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:40.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.762+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0ec00594e0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.958+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f0ec819eac0 con 0x7f0ec8100cb0 2024-02-19T19:02:40.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.958+0000 7f0ebffff700 1 -- 172.21.15.39:0/3892122770 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7f0ec0059070 con 0x7f0ec8100cb0 2024-02-19T19:02:40.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.960+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0eb406c490 msgr2=0x7f0eb406e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:40.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.960+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0eb406c490 0x7f0eb406e950 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000b5c0 tx=0x7f0eb0005e20 comp rx=0 tx=0).stop 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.960+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 msgr2=0x7f0ec8074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.960+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 0x7f0ec8074b30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0007e90 tx=0x7f0ec000d3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 shutdown_connections 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f0eb406c490 0x7f0eb406e950 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ec8100cb0 0x7f0ec8074b30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 --2- 172.21.15.39:0/3892122770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ec8101eb0 0x7f0ec8073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 >> 172.21.15.39:0/3892122770 conn(0x7f0ec80fc230 msgr2=0x7f0ec81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 shutdown_connections 2024-02-19T19:02:40.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:40.961+0000 7f0ecd3d7700 1 -- 172.21.15.39:0/3892122770 wait complete. 2024-02-19T19:02:41.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-19T19:02:41.848 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:41.969 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:41 smithi039 ceph-mon[28988]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T19:02:42.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:41 smithi086 ceph-mon[37629]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T19:02:43.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 -- 172.21.15.39:0/1032842620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c0ff540 msgr2=0x7f018c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:43.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 --2- 172.21.15.39:0/1032842620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c0ff540 0x7f018c101970 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0184009910 tx=0x7f0184009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:43.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 -- 172.21.15.39:0/1032842620 shutdown_connections 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 --2- 172.21.15.39:0/1032842620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c101eb0 0x7f018c104340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 --2- 172.21.15.39:0/1032842620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c0ff540 0x7f018c101970 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 -- 172.21.15.39:0/1032842620 >> 172.21.15.39:0/1032842620 conn(0x7f018c0fb160 msgr2=0x7f018c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 -- 172.21.15.39:0/1032842620 shutdown_connections 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 -- 172.21.15.39:0/1032842620 wait complete. 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 Processor -- start 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.551+0000 7f0193af2700 1 -- start start 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f0193af2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0ff540 0x7f018c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f0193af2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 0x7f018c19d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f0193af2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c19d900 con 0x7f018c101eb0 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f0193af2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c1a2310 con 0x7f018c0ff540 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 0x7f018c19d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 0x7f018c19d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51598/0 (socket says 172.21.15.39:51598) 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 -- 172.21.15.39:0/24211826 learned_addr learned my addr 172.21.15.39:0/24211826 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:43.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019188e700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0ff540 0x7f018c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 -- 172.21.15.39:0/24211826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0ff540 msgr2=0x7f018c19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0ff540 0x7f018c19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 -- 172.21.15.39:0/24211826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01840095f0 con 0x7f018c101eb0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f019108d700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 0x7f018c19d2e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f018000d8d0 tx=0x7f018000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01800098a0 con 0x7f018c101eb0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0180010460 con 0x7f018c101eb0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c1a2510 con 0x7f018c101eb0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.552+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f018000f5f0 con 0x7f018c101eb0 2024-02-19T19:02:43.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.553+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c1a2a30 con 0x7f018c101eb0 2024-02-19T19:02:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.554+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c196c20 con 0x7f018c101eb0 2024-02-19T19:02:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.554+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f0180009a00 con 0x7f018c101eb0 2024-02-19T19:02:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.555+0000 7f018affd700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f017406c4e0 0x7f017406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.555+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f018008b150 con 0x7f018c101eb0 2024-02-19T19:02:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.555+0000 7f019188e700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f017406c4e0 0x7f017406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.555+0000 7f019188e700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f017406c4e0 0x7f017406e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0184006010 tx=0x7f0184005ee0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:43.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.559+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f01800558b0 con 0x7f018c101eb0 2024-02-19T19:02:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.757+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f018c068af0 con 0x7f018c101eb0 2024-02-19T19:02:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.757+0000 7f018affd700 1 -- 172.21.15.39:0/24211826 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0180058ed0 con 0x7f018c101eb0 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f017406c4e0 msgr2=0x7f017406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f017406c4e0 0x7f017406e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0184006010 tx=0x7f0184005ee0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 msgr2=0x7f018c19d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 0x7f018c19d2e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f018000d8d0 tx=0x7f018000dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 shutdown_connections 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f017406c4e0 0x7f017406e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f018c0ff540 0x7f018c19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 --2- 172.21.15.39:0/24211826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f018c101eb0 0x7f018c19d2e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 >> 172.21.15.39:0/24211826 conn(0x7f018c0fb160 msgr2=0x7f018c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 shutdown_connections 2024-02-19T19:02:43.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:43.759+0000 7f0193af2700 1 -- 172.21.15.39:0/24211826 wait complete. 2024-02-19T19:02:43.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:43 smithi039 ceph-mon[28988]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-02-19T19:02:44.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:43 smithi086 ceph-mon[37629]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-02-19T19:02:44.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-19T19:02:44.660 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:45.097 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:44 smithi039 ceph-mon[28988]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-02-19T19:02:45.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:44 smithi086 ceph-mon[37629]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-02-19T19:02:46.208 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:46.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:46.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 -- 172.21.15.39:0/2738114466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ff540 msgr2=0x7f5e900ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 --2- 172.21.15.39:0/2738114466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ff540 0x7f5e900ff960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009910 tx=0x7f5e8c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 -- 172.21.15.39:0/2738114466 shutdown_connections 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 --2- 172.21.15.39:0/2738114466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e900ffea0 0x7f5e90100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 --2- 172.21.15.39:0/2738114466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ff540 0x7f5e900ff960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 -- 172.21.15.39:0/2738114466 >> 172.21.15.39:0/2738114466 conn(0x7f5e900fb160 msgr2=0x7f5e900fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 -- 172.21.15.39:0/2738114466 shutdown_connections 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 -- 172.21.15.39:0/2738114466 wait complete. 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 Processor -- start 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.485+0000 7f5e9804a700 1 -- start start 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e9804a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e900ff540 0x7f5e901989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e9804a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 0x7f5e90198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:46.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e9804a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e90199520 con 0x7f5e900ffea0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e9804a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e90199660 con 0x7f5e900ff540 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 0x7f5e90198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 0x7f5e90198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58518/0 (socket says 172.21.15.39:58518) 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 -- 172.21.15.39:0/4197753985 learned_addr learned my addr 172.21.15.39:0/4197753985 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e95de6700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e900ff540 0x7f5e901989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 -- 172.21.15.39:0/4197753985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e900ff540 msgr2=0x7f5e901989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e900ff540 0x7f5e901989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 -- 172.21.15.39:0/4197753985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c0095f0 con 0x7f5e900ffea0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e955e5700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 0x7f5e90198f00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000c8a0 tx=0x7f5e8000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e800078e0 con 0x7f5e900ffea0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5e8000f450 con 0x7f5e900ffea0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.486+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9019e110 con 0x7f5e900ffea0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.487+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e8000e5e0 con 0x7f5e900ffea0 2024-02-19T19:02:46.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.487+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9019e660 con 0x7f5e900ffea0 2024-02-19T19:02:46.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.488+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e90192820 con 0x7f5e900ffea0 2024-02-19T19:02:46.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.488+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5e80007a40 con 0x7f5e900ffea0 2024-02-19T19:02:46.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.488+0000 7f5e86ffd700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e7c06c4e0 0x7f5e7c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:46.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.488+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5e8008b850 con 0x7f5e900ffea0 2024-02-19T19:02:46.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.488+0000 7f5e95de6700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e7c06c4e0 0x7f5e7c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:46.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.489+0000 7f5e95de6700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e7c06c4e0 0x7f5e7c06e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b5c0 tx=0x7f5e8c0052a0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:46.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.493+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5e80059a60 con 0x7f5e900ffea0 2024-02-19T19:02:46.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.690+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f5e9002cd20 con 0x7f5e900ffea0 2024-02-19T19:02:46.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.690+0000 7f5e86ffd700 1 -- 172.21.15.39:0/4197753985 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f5e800595f0 con 0x7f5e900ffea0 2024-02-19T19:02:46.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e7c06c4e0 msgr2=0x7f5e7c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:46.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e7c06c4e0 0x7f5e7c06e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00b5c0 tx=0x7f5e8c0052a0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 msgr2=0x7f5e90198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:46.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 0x7f5e90198f00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000c8a0 tx=0x7f5e8000cbb0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 shutdown_connections 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f5e7c06c4e0 0x7f5e7c06e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e900ff540 0x7f5e901989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 --2- 172.21.15.39:0/4197753985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e900ffea0 0x7f5e90198f00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.692+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 >> 172.21.15.39:0/4197753985 conn(0x7f5e900fb160 msgr2=0x7f5e90107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.693+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 shutdown_connections 2024-02-19T19:02:46.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:46.693+0000 7f5e9804a700 1 -- 172.21.15.39:0/4197753985 wait complete. 2024-02-19T19:02:47.146 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:47 smithi039 ceph-mon[28988]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-02-19T19:02:47.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:47 smithi086 ceph-mon[37629]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-02-19T19:02:47.465 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-19T19:02:47.680 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:49.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 -- 172.21.15.39:0/2567180401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 msgr2=0x7fa39c100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/2567180401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c100030 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa39400b0a0 tx=0x7fa39400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 -- 172.21.15.39:0/2567180401 shutdown_connections 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/2567180401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c100e10 0x7fa39c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/2567180401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c100030 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 -- 172.21.15.39:0/2567180401 >> 172.21.15.39:0/2567180401 conn(0x7fa39c0fb190 msgr2=0x7fa39c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 -- 172.21.15.39:0/2567180401 shutdown_connections 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 -- 172.21.15.39:0/2567180401 wait complete. 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 Processor -- start 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.422+0000 7fa3a2dde700 1 -- start start 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a2dde700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c0690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a2dde700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c100e10 0x7fa39c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a2dde700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1970a0 con 0x7fa39c0ffc10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a2dde700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c19db70 con 0x7fa39c100e10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c0690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c0690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58530/0 (socket says 172.21.15.39:58530) 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 -- 172.21.15.39:0/3739074717 learned_addr learned my addr 172.21.15.39:0/3739074717 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa39bfff700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c100e10 0x7fa39c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 -- 172.21.15.39:0/3739074717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c100e10 msgr2=0x7fa39c19d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c100e10 0x7fa39c19d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 -- 172.21.15.39:0/3739074717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394009d00 con 0x7fa39c0ffc10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a0b7a700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c0690e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa394015040 tx=0x7fa394009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa39400e040 con 0x7fa39c0ffc10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c19ddc0 con 0x7fa39c0ffc10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa3940044f0 con 0x7fa39c0ffc10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.423+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa39401d920 con 0x7fa39c0ffc10 2024-02-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.424+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c19e2b0 con 0x7fa39c0ffc10 2024-02-19T19:02:49.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.424+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c108700 con 0x7fa39c0ffc10 2024-02-19T19:02:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.425+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa394004040 con 0x7fa39c0ffc10 2024-02-19T19:02:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.425+0000 7fa399ffb700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa38806c4e0 0x7fa38806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.426+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa39408d190 con 0x7fa39c0ffc10 2024-02-19T19:02:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.426+0000 7fa39bfff700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa38806c4e0 0x7fa38806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.426+0000 7fa39bfff700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa38806c4e0 0x7fa38806e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa39c069660 tx=0x7fa38c00b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.430+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa39405b3a0 con 0x7fa39c0ffc10 2024-02-19T19:02:49.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.651+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}) v1 -- 0x7fa39c105800 con 0x7fa38806c4e0 2024-02-19T19:02:49.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.661+0000 7fa399ffb700 1 -- 172.21.15.39:0/3739074717 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fa39c105800 con 0x7fa38806c4e0 2024-02-19T19:02:49.681 INFO:teuthology.orchestra.run.smithi039.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa38806c4e0 msgr2=0x7fa38806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa38806c4e0 0x7fa38806e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa39c069660 tx=0x7fa38c00b410 comp rx=0 tx=0).stop 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 msgr2=0x7fa39c0690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c0690e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa394015040 tx=0x7fa394009760 comp rx=0 tx=0).stop 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 shutdown_connections 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fa38806c4e0 0x7fa38806e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa39c0ffc10 0x7fa39c0690e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 --2- 172.21.15.39:0/3739074717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c100e10 0x7fa39c19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 >> 172.21.15.39:0/3739074717 conn(0x7fa39c0fb190 msgr2=0x7fa39c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 shutdown_connections 2024-02-19T19:02:49.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:49.663+0000 7fa3a2dde700 1 -- 172.21.15.39:0/3739074717 wait complete. 2024-02-19T19:02:49.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:49 smithi039 ceph-mon[28988]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:02:49.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:49.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:49 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:50.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:49 smithi086 ceph-mon[37629]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:02:50.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:50.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:49 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:50.436 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T19:02:50.437 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:02:50.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-19T19:02:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:02:51.094 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:50 smithi039 ceph-mon[28988]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:51.095 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:50 smithi039 ceph-mon[28988]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:02:51.095 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:51.095 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:51.095 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:50 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:51.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:50 smithi086 ceph-mon[37629]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:51.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:50 smithi086 ceph-mon[37629]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:02:51.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:51.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:51.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:50 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:02:52.046 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:51 smithi039 ceph-mon[28988]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:02:52.046 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:51 smithi039 ceph-mon[28988]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:02:52.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:51 smithi086 ceph-mon[37629]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:02:52.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:51 smithi086 ceph-mon[37629]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 -- 172.21.15.39:0/574602122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 msgr2=0x7f0020100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 --2- 172.21.15.39:0/574602122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020100030 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f001800b0a0 tx=0x7f001800b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 -- 172.21.15.39:0/574602122 shutdown_connections 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 --2- 172.21.15.39:0/574602122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020100e10 0x7f0020101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 --2- 172.21.15.39:0/574602122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020100030 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 -- 172.21.15.39:0/574602122 >> 172.21.15.39:0/574602122 conn(0x7f00200fb190 msgr2=0x7f00200fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:52.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 -- 172.21.15.39:0/574602122 shutdown_connections 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.326+0000 7f0027c51700 1 -- 172.21.15.39:0/574602122 wait complete. 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.327+0000 7f0027c51700 1 Processor -- start 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.327+0000 7f0027c51700 1 -- start start 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.327+0000 7f0027c51700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020069100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.327+0000 7f0027c51700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020100e10 0x7f002019b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.327+0000 7f0027c51700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002019b7e0 con 0x7f00200ffc10 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.327+0000 7f0027c51700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002019b950 con 0x7f0020100e10 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020069100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020069100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58546/0 (socket says 172.21.15.39:58546) 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 -- 172.21.15.39:0/2611557545 learned_addr learned my addr 172.21.15.39:0/2611557545 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00251ec700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020100e10 0x7f002019b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 -- 172.21.15.39:0/2611557545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020100e10 msgr2=0x7f002019b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020100e10 0x7f002019b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 -- 172.21.15.39:0/2611557545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018009d00 con 0x7f00200ffc10 2024-02-19T19:02:52.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f00259ed700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020069100 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00180077b0 tx=0x7f0018009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f001800e040 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0018003de0 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002019bbd0 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f001801d7f0 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.328+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002019c0c0 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.329+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0020108700 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.330+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f0018004780 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.330+0000 7f0016ffd700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f000c06c4e0 0x7f000c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.331+0000 7f00251ec700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f000c06c4e0 0x7f000c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.331+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f001808d300 con 0x7f00200ffc10 2024-02-19T19:02:52.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.331+0000 7f00251ec700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f000c06c4e0 0x7f000c06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0020069660 tx=0x7f00100073d0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:52.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.335+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f001805b5d0 con 0x7f00200ffc10 2024-02-19T19:02:52.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.564+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0020105760 con 0x7f000c06c4e0 2024-02-19T19:02:52.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.565+0000 7f0016ffd700 1 -- 172.21.15.39:0/2611557545 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f0020105760 con 0x7f000c06c4e0 2024-02-19T19:02:52.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f000c06c4e0 msgr2=0x7f000c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f000c06c4e0 0x7f000c06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0020069660 tx=0x7f00100073d0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 msgr2=0x7f0020069100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020069100 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f00180077b0 tx=0x7f0018009760 comp rx=0 tx=0).stop 2024-02-19T19:02:52.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 shutdown_connections 2024-02-19T19:02:52.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f000c06c4e0 0x7f000c06e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00200ffc10 0x7f0020069100 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 --2- 172.21.15.39:0/2611557545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020100e10 0x7f002019b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 >> 172.21.15.39:0/2611557545 conn(0x7f00200fb190 msgr2=0x7f0020104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:52.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.567+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 shutdown_connections 2024-02-19T19:02:52.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.568+0000 7f0027c51700 1 -- 172.21.15.39:0/2611557545 wait complete. 2024-02-19T19:02:52.597 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:02:52.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 -- 172.21.15.39:0/1066496290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 msgr2=0x7f78b00fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/1066496290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b00fffe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009910 tx=0x7f78a0009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:52.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 -- 172.21.15.39:0/1066496290 shutdown_connections 2024-02-19T19:02:52.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/1066496290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b0100dc0 0x7f78b0101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/1066496290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b00fffe0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 -- 172.21.15.39:0/1066496290 >> 172.21.15.39:0/1066496290 conn(0x7f78b00fb160 msgr2=0x7f78b00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:52.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 -- 172.21.15.39:0/1066496290 shutdown_connections 2024-02-19T19:02:52.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.673+0000 7f78b4bd6700 1 -- 172.21.15.39:0/1066496290 wait complete. 2024-02-19T19:02:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78b4bd6700 1 Processor -- start 2024-02-19T19:02:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78b4bd6700 1 -- start start 2024-02-19T19:02:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78b4bd6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b01967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78b4bd6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b0100dc0 0x7f78b0196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78b4bd6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b0197350 con 0x7f78b00ffbc0 2024-02-19T19:02:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78b4bd6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b0197490 con 0x7f78b0100dc0 2024-02-19T19:02:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.674+0000 7f78ae59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b01967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78ae59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b01967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58556/0 (socket says 172.21.15.39:58556) 2024-02-19T19:02:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78ae59c700 1 -- 172.21.15.39:0/3346727914 learned_addr learned my addr 172.21.15.39:0/3346727914 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:52.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78add9b700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b0100dc0 0x7f78b0196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:52.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78ae59c700 1 -- 172.21.15.39:0/3346727914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b0100dc0 msgr2=0x7f78b0196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78ae59c700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b0100dc0 0x7f78b0196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78ae59c700 1 -- 172.21.15.39:0/3346727914 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00095f0 con 0x7f78b00ffbc0 2024-02-19T19:02:52.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78ae59c700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b01967f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f78a000bda0 tx=0x7f78a000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:52.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a001d070 con 0x7f78b00ffbc0 2024-02-19T19:02:52.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b019bee0 con 0x7f78b00ffbc0 2024-02-19T19:02:52.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f78a000f460 con 0x7f78b00ffbc0 2024-02-19T19:02:52.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78a0021640 con 0x7f78b00ffbc0 2024-02-19T19:02:52.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.675+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b019c350 con 0x7f78b00ffbc0 2024-02-19T19:02:52.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.676+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b0068af0 con 0x7f78b00ffbc0 2024-02-19T19:02:52.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.677+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f78a000f5d0 con 0x7f78b00ffbc0 2024-02-19T19:02:52.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.677+0000 7f789f7fe700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f789806c4e0 0x7f789806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:52.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.677+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f78a008cb50 con 0x7f78b00ffbc0 2024-02-19T19:02:52.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.677+0000 7f78add9b700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f789806c4e0 0x7f789806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:52.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.678+0000 7f78add9b700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f789806c4e0 0x7f789806e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78a4005950 tx=0x7f78a400b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:52.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.682+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f78a005ad60 con 0x7f78b00ffbc0 2024-02-19T19:02:52.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.926+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78b00695e0 con 0x7f789806c4e0 2024-02-19T19:02:52.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.928+0000 7f789f7fe700 1 -- 172.21.15.39:0/3346727914 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f78b00695e0 con 0x7f789806c4e0 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f789806c4e0 msgr2=0x7f789806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f789806c4e0 0x7f789806e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78a4005950 tx=0x7f78a400b410 comp rx=0 tx=0).stop 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 msgr2=0x7f78b01967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b01967f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f78a000bda0 tx=0x7f78a000bdd0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 shutdown_connections 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f789806c4e0 0x7f789806e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78b00ffbc0 0x7f78b01967f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 --2- 172.21.15.39:0/3346727914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b0100dc0 0x7f78b0196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:52.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 >> 172.21.15.39:0/3346727914 conn(0x7f78b00fb160 msgr2=0x7f78b0068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:52.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 shutdown_connections 2024-02-19T19:02:52.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:52.930+0000 7f78b4bd6700 1 -- 172.21.15.39:0/3346727914 wait complete. 2024-02-19T19:02:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 -- 172.21.15.39:0/2225598781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d4100520 msgr2=0x7f65d4100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 --2- 172.21.15.39:0/2225598781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d4100520 0x7f65d4100980 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009910 tx=0x7f65c8009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 -- 172.21.15.39:0/2225598781 shutdown_connections 2024-02-19T19:02:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 --2- 172.21.15.39:0/2225598781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d4100520 0x7f65d4100980 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 --2- 172.21.15.39:0/2225598781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d40ff320 0x7f65d40ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 -- 172.21.15.39:0/2225598781 >> 172.21.15.39:0/2225598781 conn(0x7f65d40fa8c0 msgr2=0x7f65d40fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:53.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 -- 172.21.15.39:0/2225598781 shutdown_connections 2024-02-19T19:02:53.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.041+0000 7f65d8f32700 1 -- 172.21.15.39:0/2225598781 wait complete. 2024-02-19T19:02:53.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d8f32700 1 Processor -- start 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d8f32700 1 -- start start 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d8f32700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 0x7f65d41964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d8f32700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d4100520 0x7f65d41969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d8f32700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d4197000 con 0x7f65d40ff320 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d8f32700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d4197140 con 0x7f65d4100520 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 0x7f65d41964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 0x7f65d41964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58586/0 (socket says 172.21.15.39:58586) 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d259c700 1 -- 172.21.15.39:0/3040630022 learned_addr learned my addr 172.21.15.39:0/3040630022 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d1d9b700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d4100520 0x7f65d41969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d259c700 1 -- 172.21.15.39:0/3040630022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d4100520 msgr2=0x7f65d41969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d259c700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d4100520 0x7f65d41969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.042+0000 7f65d259c700 1 -- 172.21.15.39:0/3040630022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80095f0 con 0x7f65d40ff320 2024-02-19T19:02:53.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.043+0000 7f65d259c700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 0x7f65d41964a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f65c400d900 tx=0x7f65c400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.043+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c40041f0 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.043+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f65c4004350 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.043+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d419bbf0 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.043+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c4003d20 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.043+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d419c140 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.044+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d4108700 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.044+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f65c4009730 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.045+0000 7f65c37fe700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f65bc06c4e0 0x7f65bc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.045+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f65c408b710 con 0x7f65d40ff320 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.045+0000 7f65d1d9b700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f65bc06c4e0 0x7f65bc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.045+0000 7f65d1d9b700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f65bc06c4e0 0x7f65bc06e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f65c8006010 tx=0x7f65c800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:53.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.049+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f65c4059920 con 0x7f65d40ff320 2024-02-19T19:02:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.267+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f65d4106070 con 0x7f65bc06c4e0 2024-02-19T19:02:53.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.278+0000 7f65c37fe700 1 -- 172.21.15.39:0/3040630022 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f65d4106070 con 0x7f65bc06c4e0 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (4m) 23s ago 6m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (6m) 23s ago 6m 7424k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (5m) 27s ago 5m 7382k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (6m) 23s ago 6m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (5m) 27s ago 5m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (4m) 23s ago 6m 87.1M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (43s) 23s ago 43s 14.7M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (50s) 23s ago 50s 11.5M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (47s) 27s ago 47s 18.6M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (41s) 27s ago 40s 13.3M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:9283,8765,8443 running (8m) 23s ago 8m 492M - 18.2.1 94eadd990e0a f3c738bbd2c8 2024-02-19T19:02:53.298 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (5m) 27s ago 5m 443M - 18.2.1 94eadd990e0a 1b78d388267f 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (8m) 23s ago 8m 45.8M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (5m) 27s ago 5m 35.5M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (6m) 23s ago 6m 14.5M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (5m) 27s ago 5m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (4m) 23s ago 4m 50.7M 3946M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (4m) 23s ago 4m 48.2M 3946M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (3m) 23s ago 3m 46.7M 3946M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (3m) 27s ago 3m 46.2M 4970M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (2m) 27s ago 2m 43.8M 4970M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (2m) 27s ago 2m 46.0M 4970M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:02:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (4m) 23s ago 6m 55.5M - 2.43.0 a07b618ecd1d 54ef95da7db1 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f65bc06c4e0 msgr2=0x7f65bc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f65bc06c4e0 0x7f65bc06e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f65c8006010 tx=0x7f65c800b540 comp rx=0 tx=0).stop 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 msgr2=0x7f65d41964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 0x7f65d41964a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f65c400d900 tx=0x7f65c400dc10 comp rx=0 tx=0).stop 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 shutdown_connections 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f65bc06c4e0 0x7f65bc06e9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65d40ff320 0x7f65d41964a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 --2- 172.21.15.39:0/3040630022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65d4100520 0x7f65d41969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 >> 172.21.15.39:0/3040630022 conn(0x7f65d40fa8c0 msgr2=0x7f65d4105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:53.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 shutdown_connections 2024-02-19T19:02:53.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.280+0000 7f65d8f32700 1 -- 172.21.15.39:0/3040630022 wait complete. 2024-02-19T19:02:53.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 -- 172.21.15.39:0/2423916823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 msgr2=0x7fdd30103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/2423916823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30103b90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009910 tx=0x7fdd2c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:53.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 -- 172.21.15.39:0/2423916823 shutdown_connections 2024-02-19T19:02:53.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/2423916823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 0x7fdd301064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/2423916823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30103b90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 -- 172.21.15.39:0/2423916823 >> 172.21.15.39:0/2423916823 conn(0x7fdd300fb190 msgr2=0x7fdd300fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.396+0000 7fdd37e4f700 1 -- 172.21.15.39:0/2423916823 shutdown_connections 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 -- 172.21.15.39:0/2423916823 wait complete. 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 Processor -- start 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 -- start start 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 0x7fdd301969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30196fc0 con 0x7fdd301017a0 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd37e4f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30197100 con 0x7fdd301040d0 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd35beb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd35beb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58594/0 (socket says 172.21.15.39:58594) 2024-02-19T19:02:53.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.397+0000 7fdd35beb700 1 -- 172.21.15.39:0/3661927310 learned_addr learned my addr 172.21.15.39:0/3661927310 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:53.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd353ea700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 0x7fdd301969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd35beb700 1 -- 172.21.15.39:0/3661927310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 msgr2=0x7fdd301969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd35beb700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 0x7fdd301969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd35beb700 1 -- 172.21.15.39:0/3661927310 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c0095f0 con 0x7fdd301017a0 2024-02-19T19:02:53.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd353ea700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 0x7fdd301969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd35beb700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30196460 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c006010 tx=0x7fdd2c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2c01d070 con 0x7fdd301017a0 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdd2c00f460 con 0x7fdd301017a0 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3019bb50 con 0x7fdd301017a0 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd2c017630 con 0x7fdd301017a0 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.398+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3019c040 con 0x7fdd301017a0 2024-02-19T19:02:53.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.399+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd30190610 con 0x7fdd301017a0 2024-02-19T19:02:53.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.400+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdd2c003a60 con 0x7fdd301017a0 2024-02-19T19:02:53.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.400+0000 7fdd26ffd700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdd1c06c4e0 0x7fdd1c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.400+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdd2c08ccc0 con 0x7fdd301017a0 2024-02-19T19:02:53.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.400+0000 7fdd353ea700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdd1c06c4e0 0x7fdd1c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.401+0000 7fdd353ea700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdd1c06c4e0 0x7fdd1c06e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd20005fd0 tx=0x7fdd20005ee0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:53.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.405+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdd2c05af50 con 0x7fdd301017a0 2024-02-19T19:02:53.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.711+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fdd3002cc70 con 0x7fdd301017a0 2024-02-19T19:02:53.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.711+0000 7fdd26ffd700 1 -- 172.21.15.39:0/3661927310 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fdd2c05aae0 con 0x7fdd301017a0 2024-02-19T19:02:53.731 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:02:53.731 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:02:53.731 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:02:53.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:02:53.732 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:02:53.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdd1c06c4e0 msgr2=0x7fdd1c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdd1c06c4e0 0x7fdd1c06e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd20005fd0 tx=0x7fdd20005ee0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 msgr2=0x7fdd30196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30196460 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c006010 tx=0x7fdd2c00be20 comp rx=0 tx=0).stop 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 shutdown_connections 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdd1c06c4e0 0x7fdd1c06e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd301017a0 0x7fdd30196460 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 --2- 172.21.15.39:0/3661927310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd301040d0 0x7fdd301969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 >> 172.21.15.39:0/3661927310 conn(0x7fdd300fb190 msgr2=0x7fdd300fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 shutdown_connections 2024-02-19T19:02:53.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.713+0000 7fdd37e4f700 1 -- 172.21.15.39:0/3661927310 wait complete. 2024-02-19T19:02:53.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 -- 172.21.15.39:0/595822497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 msgr2=0x7ff150100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 --2- 172.21.15.39:0/595822497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150100030 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff140009910 tx=0x7ff140009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 -- 172.21.15.39:0/595822497 shutdown_connections 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 --2- 172.21.15.39:0/595822497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 0x7ff150101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 --2- 172.21.15.39:0/595822497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150100030 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 -- 172.21.15.39:0/595822497 >> 172.21.15.39:0/595822497 conn(0x7ff1500fb190 msgr2=0x7ff1500fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 -- 172.21.15.39:0/595822497 shutdown_connections 2024-02-19T19:02:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.827+0000 7ff154b3a700 1 -- 172.21.15.39:0/595822497 wait complete. 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff154b3a700 1 Processor -- start 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff154b3a700 1 -- start start 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff154b3a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff154b3a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 0x7ff15019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff154b3a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15019b760 con 0x7ff1500ffc10 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff154b3a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15019b8a0 con 0x7ff150100e10 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.828+0000 7ff14e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150069140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58614/0 (socket says 172.21.15.39:58614) 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14e59c700 1 -- 172.21.15.39:0/4281792558 learned_addr learned my addr 172.21.15.39:0/4281792558 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14dd9b700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 0x7ff15019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14e59c700 1 -- 172.21.15.39:0/4281792558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 msgr2=0x7ff15019b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14e59c700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 0x7ff15019b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14e59c700 1 -- 172.21.15.39:0/4281792558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400095f0 con 0x7ff1500ffc10 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14dd9b700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 0x7ff15019b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:02:53.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff14e59c700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150069140 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff14000bda0 tx=0x7ff14000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:53.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff14001d070 con 0x7ff1500ffc10 2024-02-19T19:02:53.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff140004c50 con 0x7ff1500ffc10 2024-02-19T19:02:53.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15019bb20 con 0x7ff1500ffc10 2024-02-19T19:02:53.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff14000f690 con 0x7ff1500ffc10 2024-02-19T19:02:53.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.829+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15019c010 con 0x7ff1500ffc10 2024-02-19T19:02:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.830+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff150108700 con 0x7ff1500ffc10 2024-02-19T19:02:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.831+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff1400229b0 con 0x7ff1500ffc10 2024-02-19T19:02:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.832+0000 7ff13f7fe700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff13806c4e0 0x7ff13806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.832+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff14008cfa0 con 0x7ff1500ffc10 2024-02-19T19:02:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.832+0000 7ff14dd9b700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff13806c4e0 0x7ff13806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.832+0000 7ff14dd9b700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff13806c4e0 0x7ff13806e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff144005fd0 tx=0x7ff144005e20 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:53.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:53.836+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff14005b230 con 0x7ff1500ffc10 2024-02-19T19:02:53.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:53 smithi039 ceph-mon[28988]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:53.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:53 smithi039 ceph-mon[28988]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:02:53.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:53 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3661927310' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.091+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff15019c6e0 con 0x7ff1500ffc10 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.092+0000 7ff13f7fe700 1 -- 172.21.15.39:0/4281792558 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1847 (secure 0 0 0) 0x7ff14005adc0 con 0x7ff1500ffc10 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:02:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:02:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:02:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:02:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff13806c4e0 msgr2=0x7ff13806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff13806c4e0 0x7ff13806e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff144005fd0 tx=0x7ff144005e20 comp rx=0 tx=0).stop 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 msgr2=0x7ff150069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150069140 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff14000bda0 tx=0x7ff14000be80 comp rx=0 tx=0).stop 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 shutdown_connections 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7ff13806c4e0 0x7ff13806e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1500ffc10 0x7ff150069140 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 --2- 172.21.15.39:0/4281792558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff150100e10 0x7ff15019b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 >> 172.21.15.39:0/4281792558 conn(0x7ff1500fb190 msgr2=0x7ff150104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 shutdown_connections 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.095+0000 7ff154b3a700 1 -- 172.21.15.39:0/4281792558 wait complete. 2024-02-19T19:02:54.116 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:02:54.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.203+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/253640455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b580ffbb0 msgr2=0x7f1b580fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.203+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/253640455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b580ffbb0 0x7f1b580fffd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009910 tx=0x7f1b48009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:54.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.203+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/253640455 shutdown_connections 2024-02-19T19:02:54.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.203+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/253640455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b58100db0 0x7f1b58101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.203+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/253640455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b580ffbb0 0x7f1b580fffd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.203+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/253640455 >> 172.21.15.39:0/253640455 conn(0x7f1b580fb150 msgr2=0x7f1b580fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:54.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.204+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/253640455 shutdown_connections 2024-02-19T19:02:54.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.204+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/253640455 wait complete. 2024-02-19T19:02:54.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.204+0000 7f1b5c8dd700 1 Processor -- start 2024-02-19T19:02:54.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.204+0000 7f1b5c8dd700 1 -- start start 2024-02-19T19:02:54.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b5c8dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b580ffbb0 0x7f1b58069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:54.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b5c8dd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 0x7f1b5819b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b5c8dd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5819b810 con 0x7f1b58100db0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b5c8dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5819b950 con 0x7f1b580ffbb0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 0x7f1b5819b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 0x7f1b5819b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58626/0 (socket says 172.21.15.39:58626) 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 -- 172.21.15.39:0/2075251985 learned_addr learned my addr 172.21.15.39:0/2075251985 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b5659c700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b580ffbb0 0x7f1b58069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 -- 172.21.15.39:0/2075251985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b580ffbb0 msgr2=0x7f1b58069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b580ffbb0 0x7f1b58069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 -- 172.21.15.39:0/2075251985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b480095f0 con 0x7f1b58100db0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b55d9b700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 0x7f1b5819b240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00d8d0 tx=0x7f1b4c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b4c0098a0 con 0x7f1b58100db0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.205+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5819bc30 con 0x7f1b58100db0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.206+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5819c180 con 0x7f1b58100db0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.206+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b4c010460 con 0x7f1b58100db0 2024-02-19T19:02:54.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.206+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b4c00f5f0 con 0x7f1b58100db0 2024-02-19T19:02:54.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.207+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38005320 con 0x7f1b58100db0 2024-02-19T19:02:54.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.207+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1b4c009a00 con 0x7f1b58100db0 2024-02-19T19:02:54.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.207+0000 7f1b477fe700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1b4006c4e0 0x7f1b4006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:54.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.208+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1b4c08b5f0 con 0x7f1b58100db0 2024-02-19T19:02:54.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.208+0000 7f1b5659c700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1b4006c4e0 0x7f1b4006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:54.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.208+0000 7f1b5659c700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1b4006c4e0 0x7f1b4006e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006010 tx=0x7f1b4800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:54.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.216+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1b4c059880 con 0x7f1b58100db0 2024-02-19T19:02:54.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:53 smithi086 ceph-mon[37629]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:54.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:53 smithi086 ceph-mon[37629]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:02:54.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:53 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3661927310' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:54.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.435+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b38000bf0 con 0x7f1b4006c4e0 2024-02-19T19:02:54.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.436+0000 7f1b477fe700 1 -- 172.21.15.39:0/2075251985 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f1b38000bf0 con 0x7f1b4006c4e0 2024-02-19T19:02:54.456 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:02:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [], 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "", 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:02:54.457 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1b4006c4e0 msgr2=0x7f1b4006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1b4006c4e0 0x7f1b4006e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006010 tx=0x7f1b4800b540 comp rx=0 tx=0).stop 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 msgr2=0x7f1b5819b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 0x7f1b5819b240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00d8d0 tx=0x7f1b4c00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 shutdown_connections 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1b4006c4e0 0x7f1b4006e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b580ffbb0 0x7f1b58069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 --2- 172.21.15.39:0/2075251985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b58100db0 0x7f1b5819b240 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 >> 172.21.15.39:0/2075251985 conn(0x7f1b580fb150 msgr2=0x7f1b58103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 shutdown_connections 2024-02-19T19:02:54.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.439+0000 7f1b5c8dd700 1 -- 172.21.15.39:0/2075251985 wait complete. 2024-02-19T19:02:54.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 -- 172.21.15.39:0/3095972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 msgr2=0x7f2d801010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 --2- 172.21.15.39:0/3095972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801010d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000b0a0 tx=0x7f2d7000b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 -- 172.21.15.39:0/3095972835 shutdown_connections 2024-02-19T19:02:54.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 --2- 172.21.15.39:0/3095972835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d80101eb0 0x7f2d80102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 --2- 172.21.15.39:0/3095972835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801010d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 -- 172.21.15.39:0/3095972835 >> 172.21.15.39:0/3095972835 conn(0x7f2d800fc230 msgr2=0x7f2d800fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:54.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 -- 172.21.15.39:0/3095972835 shutdown_connections 2024-02-19T19:02:54.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.549+0000 7f2d84a61700 1 -- 172.21.15.39:0/3095972835 wait complete. 2024-02-19T19:02:54.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d84a61700 1 Processor -- start 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d84a61700 1 -- start start 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d84a61700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d84a61700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d80101eb0 0x7f2d80196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d84a61700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80197020 con 0x7f2d80100cb0 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d84a61700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d80197160 con 0x7f2d80101eb0 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d7e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d7e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58644/0 (socket says 172.21.15.39:58644) 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d7e59c700 1 -- 172.21.15.39:0/2546028484 learned_addr learned my addr 172.21.15.39:0/2546028484 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d6ffff700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d80101eb0 0x7f2d80196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d7e59c700 1 -- 172.21.15.39:0/2546028484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d80101eb0 msgr2=0x7f2d80196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d7e59c700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d80101eb0 0x7f2d80196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.550+0000 7f2d7e59c700 1 -- 172.21.15.39:0/2546028484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d70009d00 con 0x7f2d80100cb0 2024-02-19T19:02:54.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.551+0000 7f2d7e59c700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801964c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2d700077b0 tx=0x7f2d70009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.551+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d7000e040 con 0x7f2d80100cb0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.551+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2d70003de0 con 0x7f2d80100cb0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.551+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d7001d7f0 con 0x7f2d80100cb0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.551+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8019bbb0 con 0x7f2d80100cb0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.551+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8019c0a0 con 0x7f2d80100cb0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.552+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d80109810 con 0x7f2d80100cb0 2024-02-19T19:02:54.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.552+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f2d70004780 con 0x7f2d80100cb0 2024-02-19T19:02:54.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.553+0000 7f2d6f7fe700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d6806c4e0 0x7f2d6806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:54.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.553+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2d7008d220 con 0x7f2d80100cb0 2024-02-19T19:02:54.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.553+0000 7f2d6ffff700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d6806c4e0 0x7f2d6806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:54.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.553+0000 7f2d6ffff700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d6806c4e0 0x7f2d6806e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d74005950 tx=0x7f2d7400b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:54.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.557+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2d7005b430 con 0x7f2d80100cb0 2024-02-19T19:02:54.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.862+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2d8019c770 con 0x7f2d80100cb0 2024-02-19T19:02:54.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.862+0000 7f2d6f7fe700 1 -- 172.21.15.39:0/2546028484 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f2d7005afc0 con 0x7f2d80100cb0 2024-02-19T19:02:54.882 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:02:54.882 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:02:54.882 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d6806c4e0 msgr2=0x7f2d6806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d6806c4e0 0x7f2d6806e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d74005950 tx=0x7f2d7400b410 comp rx=0 tx=0).stop 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 msgr2=0x7f2d801964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801964c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2d700077b0 tx=0x7f2d70009760 comp rx=0 tx=0).stop 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 shutdown_connections 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f2d6806c4e0 0x7f2d6806e9a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d80100cb0 0x7f2d801964c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 --2- 172.21.15.39:0/2546028484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d80101eb0 0x7f2d80196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 >> 172.21.15.39:0/2546028484 conn(0x7f2d800fc230 msgr2=0x7f2d801050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 shutdown_connections 2024-02-19T19:02:54.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:02:54.864+0000 7f2d84a61700 1 -- 172.21.15.39:0/2546028484 wait complete. 2024-02-19T19:02:55.185 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:54 smithi086 ceph-mon[37629]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:55.185 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:54 smithi086 ceph-mon[37629]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:55.185 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:54 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/4281792558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:54 smithi039 ceph-mon[28988]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:54 smithi039 ceph-mon[28988]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:54 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/4281792558' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:56.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:55 smithi039 ceph-mon[28988]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:56.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:55 smithi039 ceph-mon[28988]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:02:56.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:55 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2546028484' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:02:56.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:55 smithi086 ceph-mon[37629]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:56.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:55 smithi086 ceph-mon[37629]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:02:56.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:55 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2546028484' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:02:58.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:57 smithi086 ceph-mon[37629]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:02:58.198 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:57 smithi039 ceph-mon[28988]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:00.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:02:59 smithi039 ceph-mon[28988]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:02:59 smithi086 ceph-mon[37629]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:01.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:01.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:02.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:01 smithi039 ceph-mon[28988]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:02.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:01 smithi086 ceph-mon[37629]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:04.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:03 smithi039 ceph-mon[28988]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:03 smithi086 ceph-mon[37629]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:05.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:04 smithi086 ceph-mon[37629]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:05.434 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:04 smithi039 ceph-mon[28988]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:08.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:07 smithi039 ceph-mon[28988]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:08.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:07 smithi086 ceph-mon[37629]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:10.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:09 smithi039 ceph-mon[28988]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:10.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:09 smithi086 ceph-mon[37629]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:12.064 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:11 smithi039 ceph-mon[28988]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:12.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:11 smithi086 ceph-mon[37629]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:14.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:13 smithi039 ceph-mon[28988]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T19:03:14.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:13 smithi086 ceph-mon[37629]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T19:03:15.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:14 smithi086 ceph-mon[37629]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T19:03:15.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:14 smithi039 ceph-mon[28988]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T19:03:16.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:15 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:16.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:15 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:17.114 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:17 smithi039 ceph-mon[28988]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T19:03:17.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:17 smithi086 ceph-mon[37629]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T19:03:20.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:19 smithi039 ceph-mon[28988]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:03:20.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:19 smithi086 ceph-mon[37629]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:03:22.165 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:21 smithi039 ceph-mon[28988]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:03:22.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:21 smithi086 ceph-mon[37629]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-02-19T19:03:24.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:23 smithi039 ceph-mon[28988]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-19T19:03:24.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:23 smithi086 ceph-mon[37629]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 -- 172.21.15.39:0/2814352851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 msgr2=0x7f8860103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 --2- 172.21.15.39:0/2814352851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f8860103b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8850009910 tx=0x7f8850009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 -- 172.21.15.39:0/2814352851 shutdown_connections 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 --2- 172.21.15.39:0/2814352851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88601040a0 0x7f8860106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 --2- 172.21.15.39:0/2814352851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f8860103b60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 -- 172.21.15.39:0/2814352851 >> 172.21.15.39:0/2814352851 conn(0x7f88600fb120 msgr2=0x7f88600fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 -- 172.21.15.39:0/2814352851 shutdown_connections 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 -- 172.21.15.39:0/2814352851 wait complete. 2024-02-19T19:03:25.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 Processor -- start 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.978+0000 7f886682a700 1 -- start start 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f886682a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f88601964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f886682a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88601040a0 0x7f88601969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f886682a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8860197000 con 0x7f8860101770 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f886682a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8860197140 con 0x7f88601040a0 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f88601964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f88601964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60272/0 (socket says 172.21.15.39:60272) 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 -- 172.21.15.39:0/3662582251 learned_addr learned my addr 172.21.15.39:0/3662582251 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885f7fe700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88601040a0 0x7f88601969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 -- 172.21.15.39:0/3662582251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88601040a0 msgr2=0x7f88601969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88601040a0 0x7f88601969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 -- 172.21.15.39:0/3662582251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500095f0 con 0x7f8860101770 2024-02-19T19:03:25.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.979+0000 7f885ffff700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f88601964a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f885000b5c0 tx=0x7f885000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:25.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.980+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f885001d070 con 0x7f8860101770 2024-02-19T19:03:25.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.980+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886019bb90 con 0x7f8860101770 2024-02-19T19:03:25.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.980+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f885000f460 con 0x7f8860101770 2024-02-19T19:03:25.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.980+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8850021640 con 0x7f8860101770 2024-02-19T19:03:25.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.980+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886019c080 con 0x7f8860101770 2024-02-19T19:03:25.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.981+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8860190610 con 0x7f8860101770 2024-02-19T19:03:25.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.981+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f885000f5d0 con 0x7f8860101770 2024-02-19T19:03:25.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.982+0000 7f885d7fa700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f884806c4e0 0x7f884806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.982+0000 7f885f7fe700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f884806c4e0 0x7f884806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.982+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f885008cbe0 con 0x7f8860101770 2024-02-19T19:03:25.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.982+0000 7f885f7fe700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f884806c4e0 0x7f884806e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8854005950 tx=0x7f88540058e0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:25.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:24.987+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f885005adf0 con 0x7f8860101770 2024-02-19T19:03:25.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.235+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f88600611d0 con 0x7f884806c4e0 2024-02-19T19:03:25.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:24 smithi039 ceph-mon[28988]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-19T19:03:25.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.236+0000 7f885d7fa700 1 -- 172.21.15.39:0/3662582251 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f88600611d0 con 0x7f884806c4e0 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f884806c4e0 msgr2=0x7f884806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f884806c4e0 0x7f884806e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8854005950 tx=0x7f88540058e0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 msgr2=0x7f88601964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f88601964a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f885000b5c0 tx=0x7f885000be20 comp rx=0 tx=0).stop 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 shutdown_connections 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f884806c4e0 0x7f884806e9a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8860101770 0x7f88601964a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 --2- 172.21.15.39:0/3662582251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88601040a0 0x7f88601969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.238+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 >> 172.21.15.39:0/3662582251 conn(0x7f88600fb120 msgr2=0x7f88600fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.239+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 shutdown_connections 2024-02-19T19:03:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.239+0000 7f886682a700 1 -- 172.21.15.39:0/3662582251 wait complete. 2024-02-19T19:03:25.271 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:03:25.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:24 smithi086 ceph-mon[37629]: pgmap v226: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-19T19:03:25.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 -- 172.21.15.39:0/4136757856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42680ffc10 msgr2=0x7f4268100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 --2- 172.21.15.39:0/4136757856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42680ffc10 0x7f4268100030 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4258009910 tx=0x7f4258009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 -- 172.21.15.39:0/4136757856 shutdown_connections 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 --2- 172.21.15.39:0/4136757856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4268100e10 0x7f4268101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 --2- 172.21.15.39:0/4136757856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42680ffc10 0x7f4268100030 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 -- 172.21.15.39:0/4136757856 >> 172.21.15.39:0/4136757856 conn(0x7f42680fb190 msgr2=0x7f42680fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 -- 172.21.15.39:0/4136757856 shutdown_connections 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.365+0000 7f426e4e4700 1 -- 172.21.15.39:0/4136757856 wait complete. 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f426e4e4700 1 Processor -- start 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f426e4e4700 1 -- start start 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f426e4e4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42680ffc10 0x7f4268069100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f426e4e4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 0x7f426819b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f426e4e4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426819b7e0 con 0x7f4268100e10 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f426e4e4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426819b950 con 0x7f42680ffc10 2024-02-19T19:03:25.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f42677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 0x7f426819b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f42677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 0x7f426819b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60292/0 (socket says 172.21.15.39:60292) 2024-02-19T19:03:25.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.366+0000 7f42677fe700 1 -- 172.21.15.39:0/2546714653 learned_addr learned my addr 172.21.15.39:0/2546714653 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:25.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f4267fff700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42680ffc10 0x7f4268069100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42677fe700 1 -- 172.21.15.39:0/2546714653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42680ffc10 msgr2=0x7f4268069100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42677fe700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42680ffc10 0x7f4268069100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42677fe700 1 -- 172.21.15.39:0/2546714653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42580095f0 con 0x7f4268100e10 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42677fe700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 0x7f426819b210 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f425c00d8d0 tx=0x7f425c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425c009940 con 0x7f4268100e10 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f425c010460 con 0x7f4268100e10 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426819bc30 con 0x7f4268100e10 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425c00f5f0 con 0x7f4268100e10 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.367+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426819c130 con 0x7f4268100e10 2024-02-19T19:03:25.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.369+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4268108700 con 0x7f4268100e10 2024-02-19T19:03:25.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.369+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f425c0105d0 con 0x7f4268100e10 2024-02-19T19:03:25.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.369+0000 7f42657fa700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f425006c4e0 0x7f425006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.370+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f425c08b890 con 0x7f4268100e10 2024-02-19T19:03:25.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.370+0000 7f4267fff700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f425006c4e0 0x7f425006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.370+0000 7f4267fff700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f425006c4e0 0x7f425006e9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4258006010 tx=0x7f425800b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:25.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.378+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f425c059aa0 con 0x7f4268100e10 2024-02-19T19:03:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.630+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4268105760 con 0x7f425006c4e0 2024-02-19T19:03:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.631+0000 7f42657fa700 1 -- 172.21.15.39:0/2546714653 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f4268105760 con 0x7f425006c4e0 2024-02-19T19:03:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f425006c4e0 msgr2=0x7f425006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f425006c4e0 0x7f425006e9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4258006010 tx=0x7f425800b540 comp rx=0 tx=0).stop 2024-02-19T19:03:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 msgr2=0x7f426819b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 0x7f426819b210 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f425c00d8d0 tx=0x7f425c00dc90 comp rx=0 tx=0).stop 2024-02-19T19:03:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 shutdown_connections 2024-02-19T19:03:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f425006c4e0 0x7f425006e9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42680ffc10 0x7f4268069100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 --2- 172.21.15.39:0/2546714653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4268100e10 0x7f426819b210 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 >> 172.21.15.39:0/2546714653 conn(0x7f42680fb190 msgr2=0x7f4268104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:25.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.634+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 shutdown_connections 2024-02-19T19:03:25.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.635+0000 7f426e4e4700 1 -- 172.21.15.39:0/2546714653 wait complete. 2024-02-19T19:03:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 -- 172.21.15.39:0/2456526598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 msgr2=0x7fd61c0ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 --2- 172.21.15.39:0/2456526598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c0ff380 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009910 tx=0x7fd60c009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 -- 172.21.15.39:0/2456526598 shutdown_connections 2024-02-19T19:03:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 --2- 172.21.15.39:0/2456526598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 0x7fd61c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 --2- 172.21.15.39:0/2456526598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c0ff380 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 -- 172.21.15.39:0/2456526598 >> 172.21.15.39:0/2456526598 conn(0x7fd61c0fa8c0 msgr2=0x7fd61c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:25.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 -- 172.21.15.39:0/2456526598 shutdown_connections 2024-02-19T19:03:25.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.749+0000 7fd620c0c700 1 -- 172.21.15.39:0/2456526598 wait complete. 2024-02-19T19:03:25.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.750+0000 7fd620c0c700 1 Processor -- start 2024-02-19T19:03:25.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.750+0000 7fd620c0c700 1 -- start start 2024-02-19T19:03:25.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.750+0000 7fd620c0c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.750+0000 7fd620c0c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 0x7fd61c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.750+0000 7fd620c0c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c197030 con 0x7fd61c0fef60 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.750+0000 7fd620c0c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c197170 con 0x7fd61c101f40 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60318/0 (socket says 172.21.15.39:60318) 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 -- 172.21.15.39:0/586738955 learned_addr learned my addr 172.21.15.39:0/586738955 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd619d9b700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 0x7fd61c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 -- 172.21.15.39:0/586738955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 msgr2=0x7fd61c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 0x7fd61c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 -- 172.21.15.39:0/586738955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c0095f0 con 0x7fd61c0fef60 2024-02-19T19:03:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd619d9b700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 0x7fd61c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:03:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd61a59c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c1964d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000c00 tx=0x7fd60c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60c01d070 con 0x7fd61c0fef60 2024-02-19T19:03:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd60c005dc0 con 0x7fd61c0fef60 2024-02-19T19:03:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.751+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60c0226a0 con 0x7fd61c0fef60 2024-02-19T19:03:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.752+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c19bbc0 con 0x7fd61c0fef60 2024-02-19T19:03:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.752+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c19c080 con 0x7fd61c0fef60 2024-02-19T19:03:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.753+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c109800 con 0x7fd61c0fef60 2024-02-19T19:03:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.754+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd60c003910 con 0x7fd61c0fef60 2024-02-19T19:03:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.754+0000 7fd60b7fe700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd60406c4e0 0x7fd60406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:25.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.755+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd60c08cdc0 con 0x7fd61c0fef60 2024-02-19T19:03:25.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.755+0000 7fd619d9b700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd60406c4e0 0x7fd60406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:25.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.755+0000 7fd619d9b700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd60406c4e0 0x7fd60406e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd610009be0 tx=0x7fd610009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:25.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:25.760+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd60c05b050 con 0x7fd61c0fef60 2024-02-19T19:03:26.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.015+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd61c106890 con 0x7fd60406c4e0 2024-02-19T19:03:26.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.026+0000 7fd60b7fe700 1 -- 172.21.15.39:0/586738955 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fd61c106890 con 0x7fd60406c4e0 2024-02-19T19:03:26.046 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (5m) 55s ago 7m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (7m) 55s ago 7m 7424k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (6m) 60s ago 6m 7382k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (7m) 55s ago 7m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (6m) 60s ago 6m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (5m) 55s ago 6m 87.1M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (76s) 55s ago 76s 14.7M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (83s) 55s ago 83s 11.5M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (80s) 60s ago 80s 18.6M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (73s) 60s ago 73s 13.3M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:9283,8765,8443 running (8m) 55s ago 8m 492M - 18.2.1 94eadd990e0a f3c738bbd2c8 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (5m) 60s ago 5m 443M - 18.2.1 94eadd990e0a 1b78d388267f 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (8m) 55s ago 8m 45.8M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:03:26.047 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (5m) 60s ago 5m 35.5M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (7m) 55s ago 7m 14.5M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (6m) 60s ago 6m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (5m) 55s ago 5m 50.7M 3946M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (4m) 55s ago 4m 48.2M 3946M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (4m) 55s ago 4m 46.7M 3946M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (3m) 60s ago 3m 46.2M 4970M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (3m) 60s ago 3m 43.8M 4970M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (2m) 60s ago 2m 46.0M 4970M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:03:26.048 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (5m) 55s ago 6m 55.5M - 2.43.0 a07b618ecd1d 54ef95da7db1 2024-02-19T19:03:26.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd60406c4e0 msgr2=0x7fd60406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd60406c4e0 0x7fd60406e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd610009be0 tx=0x7fd610009260 comp rx=0 tx=0).stop 2024-02-19T19:03:26.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 msgr2=0x7fd61c1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c1964d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000c00 tx=0x7fd60c00f740 comp rx=0 tx=0).stop 2024-02-19T19:03:26.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 shutdown_connections 2024-02-19T19:03:26.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd60406c4e0 0x7fd60406e9a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd61c0fef60 0x7fd61c1964d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 --2- 172.21.15.39:0/586738955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61c101f40 0x7fd61c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 >> 172.21.15.39:0/586738955 conn(0x7fd61c0fa8c0 msgr2=0x7fd61c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:26.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 shutdown_connections 2024-02-19T19:03:26.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.029+0000 7fd620c0c700 1 -- 172.21.15.39:0/586738955 wait complete. 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 -- 172.21.15.39:0/2459797749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 msgr2=0x7fb3180ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 --2- 172.21.15.39:0/2459797749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb3180ff740 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb314009910 tx=0x7fb314009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 -- 172.21.15.39:0/2459797749 shutdown_connections 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 --2- 172.21.15.39:0/2459797749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 0x7fb318100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 --2- 172.21.15.39:0/2459797749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb3180ff740 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 -- 172.21.15.39:0/2459797749 >> 172.21.15.39:0/2459797749 conn(0x7fb3180fa8c0 msgr2=0x7fb3180fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 -- 172.21.15.39:0/2459797749 shutdown_connections 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.137+0000 7fb31fd36700 1 -- 172.21.15.39:0/2459797749 wait complete. 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31fd36700 1 Processor -- start 2024-02-19T19:03:26.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31fd36700 1 -- start start 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31fd36700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb318106320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31fd36700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 0x7fb31819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31fd36700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31819b7f0 con 0x7fb3180ff320 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31fd36700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31819b930 con 0x7fb318100520 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31dad2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb318106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31dad2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb318106320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60344/0 (socket says 172.21.15.39:60344) 2024-02-19T19:03:26.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.138+0000 7fb31dad2700 1 -- 172.21.15.39:0/288043144 learned_addr learned my addr 172.21.15.39:0/288043144 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31d2d1700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 0x7fb31819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31dad2700 1 -- 172.21.15.39:0/288043144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 msgr2=0x7fb31819b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31dad2700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 0x7fb31819b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31dad2700 1 -- 172.21.15.39:0/288043144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140095f0 con 0x7fb3180ff320 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31d2d1700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 0x7fb31819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31dad2700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb318106320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb31400bda0 tx=0x7fb31400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb31401d070 con 0x7fb3180ff320 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31819bbb0 con 0x7fb3180ff320 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb314004c50 con 0x7fb3180ff320 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb31400f690 con 0x7fb3180ff320 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.139+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31819c0a0 con 0x7fb3180ff320 2024-02-19T19:03:26.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.140+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb318108700 con 0x7fb3180ff320 2024-02-19T19:03:26.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.141+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb3140229b0 con 0x7fb3180ff320 2024-02-19T19:03:26.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.141+0000 7fb30effd700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb30406c4e0 0x7fb30406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.141+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb31408cf00 con 0x7fb3180ff320 2024-02-19T19:03:26.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.141+0000 7fb31d2d1700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb30406c4e0 0x7fb30406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.141+0000 7fb31d2d1700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb30406c4e0 0x7fb30406e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb308005fd0 tx=0x7fb308005dc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:26.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.146+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb31405b1c0 con 0x7fb3180ff320 2024-02-19T19:03:26.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:26 smithi086 ceph-mon[37629]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:26.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:26 smithi086 ceph-mon[37629]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:26.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.433+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb3181958e0 con 0x7fb3180ff320 2024-02-19T19:03:26.453 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:26 smithi039 ceph-mon[28988]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:26.453 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:26 smithi039 ceph-mon[28988]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:26.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.433+0000 7fb30effd700 1 -- 172.21.15.39:0/288043144 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fb31405ad50 con 0x7fb3180ff320 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:03:26.454 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:03:26.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:26.455 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:03:26.455 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:03:26.455 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:03:26.455 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb30406c4e0 msgr2=0x7fb30406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb30406c4e0 0x7fb30406e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb308005fd0 tx=0x7fb308005dc0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 msgr2=0x7fb318106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb318106320 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb31400bda0 tx=0x7fb31400be80 comp rx=0 tx=0).stop 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 shutdown_connections 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb30406c4e0 0x7fb30406e9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3180ff320 0x7fb318106320 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 --2- 172.21.15.39:0/288043144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb318100520 0x7fb31819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 >> 172.21.15.39:0/288043144 conn(0x7fb3180fa8c0 msgr2=0x7fb318103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 shutdown_connections 2024-02-19T19:03:26.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.435+0000 7fb31fd36700 1 -- 172.21.15.39:0/288043144 wait complete. 2024-02-19T19:03:26.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 -- 172.21.15.39:0/3167015415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 msgr2=0x7f342c1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 --2- 172.21.15.39:0/3167015415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c1010d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3414009910 tx=0x7f3414009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:26.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 -- 172.21.15.39:0/3167015415 shutdown_connections 2024-02-19T19:03:26.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 --2- 172.21.15.39:0/3167015415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f342c101eb0 0x7f342c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 --2- 172.21.15.39:0/3167015415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c1010d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 -- 172.21.15.39:0/3167015415 >> 172.21.15.39:0/3167015415 conn(0x7f342c0fc230 msgr2=0x7f342c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:26.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 -- 172.21.15.39:0/3167015415 shutdown_connections 2024-02-19T19:03:26.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.541+0000 7f3431033700 1 -- 172.21.15.39:0/3167015415 wait complete. 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f3431033700 1 Processor -- start 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f3431033700 1 -- start start 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f3431033700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f3431033700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f342c101eb0 0x7f342c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f3431033700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342c0736c0 con 0x7f342c100cb0 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f3431033700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342c073800 con 0x7f342c101eb0 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47938/0 (socket says 172.21.15.39:47938) 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342a59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f342c101eb0 0x7f342c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342ad9d700 1 -- 172.21.15.39:0/1580662484 learned_addr learned my addr 172.21.15.39:0/1580662484 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342ad9d700 1 -- 172.21.15.39:0/1580662484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f342c101eb0 msgr2=0x7f342c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342ad9d700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f342c101eb0 0x7f342c073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.542+0000 7f342ad9d700 1 -- 172.21.15.39:0/1580662484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34140095f0 con 0x7f342c100cb0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.543+0000 7f342ad9d700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c074b30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f341400bda0 tx=0x7f341400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.543+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f341401d070 con 0x7f342c100cb0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.543+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f341400f460 con 0x7f342c100cb0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.543+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342c073a80 con 0x7f342c100cb0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.543+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3414021620 con 0x7f342c100cb0 2024-02-19T19:03:26.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.543+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342c073f70 con 0x7f342c100cb0 2024-02-19T19:03:26.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.544+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342c109810 con 0x7f342c100cb0 2024-02-19T19:03:26.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.544+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f341400f5d0 con 0x7f342c100cb0 2024-02-19T19:03:26.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.545+0000 7f3423fff700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f341806c4e0 0x7f341806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.545+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f341408cb10 con 0x7f342c100cb0 2024-02-19T19:03:26.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.545+0000 7f342a59c700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f341806c4e0 0x7f341806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.545+0000 7f342a59c700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f341806c4e0 0x7f341806e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f341c005950 tx=0x7f341c00b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:26.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.549+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f341405ad20 con 0x7f342c100cb0 2024-02-19T19:03:26.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.793+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f342c109a00 con 0x7f342c100cb0 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.793+0000 7f3423fff700 1 -- 172.21.15.39:0/1580662484 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1847 (secure 0 0 0) 0x7f341405a8b0 con 0x7f342c100cb0 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:03:26.814 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:03:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:26.816 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:26.817 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:26.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f341806c4e0 msgr2=0x7f341806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f341806c4e0 0x7f341806e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f341c005950 tx=0x7f341c00b410 comp rx=0 tx=0).stop 2024-02-19T19:03:26.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 msgr2=0x7f342c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c074b30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f341400bda0 tx=0x7f341400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 shutdown_connections 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f341806c4e0 0x7f341806e9a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f342c100cb0 0x7f342c074b30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 --2- 172.21.15.39:0/1580662484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f342c101eb0 0x7f342c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.795+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 >> 172.21.15.39:0/1580662484 conn(0x7f342c0fc230 msgr2=0x7f342c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.796+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 shutdown_connections 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.796+0000 7f3431033700 1 -- 172.21.15.39:0/1580662484 wait complete. 2024-02-19T19:03:26.818 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:03:26.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 -- 172.21.15.39:0/174060240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 msgr2=0x7f1c640fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 --2- 172.21.15.39:0/174060240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c640fffc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1c54009910 tx=0x7f1c54009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:26.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 -- 172.21.15.39:0/174060240 shutdown_connections 2024-02-19T19:03:26.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 --2- 172.21.15.39:0/174060240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64100da0 0x7f1c64101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 --2- 172.21.15.39:0/174060240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c640fffc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 -- 172.21.15.39:0/174060240 >> 172.21.15.39:0/174060240 conn(0x7f1c640fb120 msgr2=0x7f1c640fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:26.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 -- 172.21.15.39:0/174060240 shutdown_connections 2024-02-19T19:03:26.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.910+0000 7f1c68974700 1 -- 172.21.15.39:0/174060240 wait complete. 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c68974700 1 Processor -- start 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c68974700 1 -- start start 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c68974700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c64069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c68974700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64100da0 0x7f1c6419b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c68974700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6419b7e0 con 0x7f1c640ffba0 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c68974700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6419b950 con 0x7f1c64100da0 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c6259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c64069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c6259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c64069990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47962/0 (socket says 172.21.15.39:47962) 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.911+0000 7f1c6259c700 1 -- 172.21.15.39:0/2561760898 learned_addr learned my addr 172.21.15.39:0/2561760898 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c5bfff700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64100da0 0x7f1c6419b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c6259c700 1 -- 172.21.15.39:0/2561760898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64100da0 msgr2=0x7f1c6419b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:26.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c6259c700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64100da0 0x7f1c6419b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:26.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c6259c700 1 -- 172.21.15.39:0/2561760898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c540095f0 con 0x7f1c640ffba0 2024-02-19T19:03:26.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c6259c700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c64069990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400bda0 tx=0x7f1c5400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:26.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5401d070 con 0x7f1c640ffba0 2024-02-19T19:03:26.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c5400f460 con 0x7f1c640ffba0 2024-02-19T19:03:26.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c54021640 con 0x7f1c640ffba0 2024-02-19T19:03:26.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6419bbd0 con 0x7f1c640ffba0 2024-02-19T19:03:26.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.912+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6419c0c0 con 0x7f1c640ffba0 2024-02-19T19:03:26.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.913+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c64108700 con 0x7f1c640ffba0 2024-02-19T19:03:26.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.914+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1c5400f5d0 con 0x7f1c640ffba0 2024-02-19T19:03:26.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.914+0000 7f1c5b7fe700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1c5006c4e0 0x7f1c5006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:26.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.914+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1c5408ca50 con 0x7f1c640ffba0 2024-02-19T19:03:26.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.915+0000 7f1c5bfff700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1c5006c4e0 0x7f1c5006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:26.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.915+0000 7f1c5bfff700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1c5006c4e0 0x7f1c5006e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c64069ef0 tx=0x7f1c4c00b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:26.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:26.919+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1c5405ace0 con 0x7f1c640ffba0 2024-02-19T19:03:27.081 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:27 smithi039 ceph-mon[28988]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:27.081 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:27 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/288043144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:27.081 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:27 smithi039 ceph-mon[28988]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-02-19T19:03:27.081 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:27 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1580662484' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:03:27.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.144+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c641056f0 con 0x7f1c5006c4e0 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.145+0000 7f1c5b7fe700 1 -- 172.21.15.39:0/2561760898 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f1c641056f0 con 0x7f1c5006c4e0 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [], 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "", 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:03:27.165 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1c5006c4e0 msgr2=0x7f1c5006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1c5006c4e0 0x7f1c5006e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c64069ef0 tx=0x7f1c4c00b410 comp rx=0 tx=0).stop 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 msgr2=0x7f1c64069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c64069990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400bda0 tx=0x7f1c5400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 shutdown_connections 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f1c5006c4e0 0x7f1c5006e9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c640ffba0 0x7f1c64069990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 --2- 172.21.15.39:0/2561760898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64100da0 0x7f1c6419b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 >> 172.21.15.39:0/2561760898 conn(0x7f1c640fb120 msgr2=0x7f1c64103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:27.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 shutdown_connections 2024-02-19T19:03:27.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.147+0000 7f1c68974700 1 -- 172.21.15.39:0/2561760898 wait complete. 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 -- 172.21.15.39:0/1108410682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8104060 msgr2=0x7f62a8106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 --2- 172.21.15.39:0/1108410682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8104060 0x7f62a8106450 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6298009910 tx=0x7f6298009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 -- 172.21.15.39:0/1108410682 shutdown_connections 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 --2- 172.21.15.39:0/1108410682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8104060 0x7f62a8106450 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 --2- 172.21.15.39:0/1108410682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62a8101730 0x7f62a8103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 -- 172.21.15.39:0/1108410682 >> 172.21.15.39:0/1108410682 conn(0x7f62a80fb120 msgr2=0x7f62a80fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:27.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 -- 172.21.15.39:0/1108410682 shutdown_connections 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.258+0000 7f62b056b700 1 -- 172.21.15.39:0/1108410682 wait complete. 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62b056b700 1 Processor -- start 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62b056b700 1 -- start start 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62b056b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 0x7f62a81964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62b056b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62a8104060 0x7f62a81969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62b056b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a8197000 con 0x7f62a8101730 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62b056b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a8197140 con 0x7f62a8104060 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62ae307700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 0x7f62a81964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62ae307700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 0x7f62a81964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47982/0 (socket says 172.21.15.39:47982) 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.259+0000 7f62ae307700 1 -- 172.21.15.39:0/3595225498 learned_addr learned my addr 172.21.15.39:0/3595225498 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62ae307700 1 -- 172.21.15.39:0/3595225498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62a8104060 msgr2=0x7f62a81969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62adb06700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62a8104060 0x7f62a81969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:27.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62ae307700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62a8104060 0x7f62a81969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62ae307700 1 -- 172.21.15.39:0/3595225498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980095f0 con 0x7f62a8101730 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62ae307700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 0x7f62a81964a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c8a0 tx=0x7f62a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a400cea0 con 0x7f62a8101730 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a819bbf0 con 0x7f62a8101730 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f62a4004830 con 0x7f62a8101730 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a40055b0 con 0x7f62a8101730 2024-02-19T19:03:27.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.260+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a819c140 con 0x7f62a8101730 2024-02-19T19:03:27.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.261+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a8190610 con 0x7f62a8101730 2024-02-19T19:03:27.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.262+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f62a4007ad0 con 0x7f62a8101730 2024-02-19T19:03:27.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.262+0000 7f629f7fe700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f629406c4e0 0x7f629406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:27.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.262+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f62a408b5e0 con 0x7f62a8101730 2024-02-19T19:03:27.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.262+0000 7f62adb06700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f629406c4e0 0x7f629406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:27.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.263+0000 7f62adb06700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f629406c4e0 0x7f629406e9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f629800b5c0 tx=0x7f6298005960 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:27.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:27 smithi086 ceph-mon[37629]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:27.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:27 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/288043144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:27.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:27 smithi086 ceph-mon[37629]: pgmap v227: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-02-19T19:03:27.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:27 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1580662484' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:03:27.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.267+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f62a40597f0 con 0x7f62a8101730 2024-02-19T19:03:27.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.556+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f62a8061a40 con 0x7f62a8101730 2024-02-19T19:03:27.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.556+0000 7f629f7fe700 1 -- 172.21.15.39:0/3595225498 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f62a4059380 con 0x7f62a8101730 2024-02-19T19:03:27.576 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:27.576 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:27.577 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f629406c4e0 msgr2=0x7f629406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f629406c4e0 0x7f629406e9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f629800b5c0 tx=0x7f6298005960 comp rx=0 tx=0).stop 2024-02-19T19:03:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 msgr2=0x7f62a81964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 0x7f62a81964a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c8a0 tx=0x7f62a400cc60 comp rx=0 tx=0).stop 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 shutdown_connections 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f629406c4e0 0x7f629406e9a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62a8101730 0x7f62a81964a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 --2- 172.21.15.39:0/3595225498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62a8104060 0x7f62a81969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 >> 172.21.15.39:0/3595225498 conn(0x7f62a80fb120 msgr2=0x7f62a80fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 shutdown_connections 2024-02-19T19:03:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:27.558+0000 7f62b056b700 1 -- 172.21.15.39:0/3595225498 wait complete. 2024-02-19T19:03:28.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:28 smithi086 ceph-mon[37629]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:28.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:28 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3595225498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:03:28.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:28 smithi039 ceph-mon[28988]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:28.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:28 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3595225498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: Upgrade: Need to upgrade myself (mgr.smithi039.ydmpuh) 2024-02-19T19:03:29.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi086 2024-02-19T19:03:29.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:29 smithi039 ceph-mon[28988]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: Upgrade: Need to upgrade myself (mgr.smithi039.ydmpuh) 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi086 2024-02-19T19:03:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:29 smithi086 ceph-mon[37629]: pgmap v228: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-02-19T19:03:31.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:31.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:32.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:31 smithi039 ceph-mon[28988]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:32.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:31 smithi086 ceph-mon[37629]: pgmap v229: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:34.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:33 smithi039 ceph-mon[28988]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:34.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:33 smithi086 ceph-mon[37629]: pgmap v230: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:35.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:34 smithi086 ceph-mon[37629]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:35.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:34 smithi039 ceph-mon[28988]: pgmap v231: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:38.117 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:37 smithi086 ceph-mon[37629]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:38.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:37 smithi039 ceph-mon[28988]: pgmap v232: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:40.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:39 smithi039 ceph-mon[28988]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:40.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:39 smithi086 ceph-mon[37629]: pgmap v233: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:42.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:41 smithi039 ceph-mon[28988]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:42.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:41 smithi086 ceph-mon[37629]: pgmap v234: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:44.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:43 smithi039 ceph-mon[28988]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:44.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:43 smithi086 ceph-mon[37629]: pgmap v235: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:44 smithi086 ceph-mon[37629]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:45.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:44 smithi039 ceph-mon[28988]: pgmap v236: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:46.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:45 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:46.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:45 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:47.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:47 smithi039 ceph-mon[28988]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:47.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:47 smithi086 ceph-mon[37629]: pgmap v237: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:50.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:49 smithi039 ceph-mon[28988]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:50.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:49 smithi086 ceph-mon[37629]: pgmap v238: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:52.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:51 smithi039 ceph-mon[28988]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:52.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:51 smithi086 ceph-mon[37629]: pgmap v239: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:54.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:53 smithi039 ceph-mon[28988]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:54.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:53 smithi086 ceph-mon[37629]: pgmap v240: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:55.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:54 smithi086 ceph-mon[37629]: pgmap v241: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:55.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:54 smithi039 ceph-mon[28988]: pgmap v241: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:57.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 -- 172.21.15.39:0/1322876801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 msgr2=0x7fcef80ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1322876801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef80ff960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcee800b0a0 tx=0x7fcee800b3b0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 -- 172.21.15.39:0/1322876801 shutdown_connections 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1322876801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 0x7fcef8100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1322876801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef80ff960 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 -- 172.21.15.39:0/1322876801 >> 172.21.15.39:0/1322876801 conn(0x7fcef80fb160 msgr2=0x7fcef80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 -- 172.21.15.39:0/1322876801 shutdown_connections 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 -- 172.21.15.39:0/1322876801 wait complete. 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.674+0000 7fceff5ae700 1 Processor -- start 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fceff5ae700 1 -- start start 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fceff5ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef81989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fceff5ae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 0x7fcef8198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:57.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fceff5ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8199520 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fceff5ae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8199660 con 0x7fcef80ffea0 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef81989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef81989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41866/0 (socket says 172.21.15.39:41866) 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 -- 172.21.15.39:0/1858572674 learned_addr learned my addr 172.21.15.39:0/1858572674 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefcb49700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 0x7fcef8198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 -- 172.21.15.39:0/1858572674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 msgr2=0x7fcef8198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 0x7fcef8198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 -- 172.21.15.39:0/1858572674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee8009d00 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefcb49700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 0x7fcef8198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcefd34a700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef81989c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcee80077b0 tx=0x7fcee8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.675+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcee800e040 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.676+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcee800ce60 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.676+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcee801daf0 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.676+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef819e0b0 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.676+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef819e5a0 con 0x7fcef80ff540 2024-02-19T19:03:57.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.676+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef8192820 con 0x7fcef80ff540 2024-02-19T19:03:57.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.677+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fcee8004250 con 0x7fcef80ff540 2024-02-19T19:03:57.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.677+0000 7fcef67fc700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcee006c4e0 0x7fcee006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:57.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.677+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcee808d2b0 con 0x7fcef80ff540 2024-02-19T19:03:57.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.677+0000 7fcefcb49700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcee006c4e0 0x7fcee006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:57.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.678+0000 7fcefcb49700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcee006c4e0 0x7fcee006e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fceec005fd0 tx=0x7fceec005dc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:57.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.682+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcee805b540 con 0x7fcef80ff540 2024-02-19T19:03:57.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.908+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcef80611d0 con 0x7fcee006c4e0 2024-02-19T19:03:57.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.909+0000 7fcef67fc700 1 -- 172.21.15.39:0/1858572674 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fcef80611d0 con 0x7fcee006c4e0 2024-02-19T19:03:57.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcee006c4e0 msgr2=0x7fcee006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:57.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcee006c4e0 0x7fcee006e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fceec005fd0 tx=0x7fceec005dc0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 msgr2=0x7fcef81989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef81989c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcee80077b0 tx=0x7fcee8003ce0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 shutdown_connections 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fcee006c4e0 0x7fcee006e9a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcef80ff540 0x7fcef81989c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 --2- 172.21.15.39:0/1858572674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcef80ffea0 0x7fcef8198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 >> 172.21.15.39:0/1858572674 conn(0x7fcef80fb160 msgr2=0x7fcef8107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 shutdown_connections 2024-02-19T19:03:57.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:57.911+0000 7fceff5ae700 1 -- 172.21.15.39:0/1858572674 wait complete. 2024-02-19T19:03:57.941 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 -- 172.21.15.39:0/2826865156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 msgr2=0x7fdbb4102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 --2- 172.21.15.39:0/2826865156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4102300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009910 tx=0x7fdba8009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 -- 172.21.15.39:0/2826865156 shutdown_connections 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 --2- 172.21.15.39:0/2826865156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4102300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 --2- 172.21.15.39:0/2826865156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbb4100ca0 0x7fdbb41010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 -- 172.21.15.39:0/2826865156 >> 172.21.15.39:0/2826865156 conn(0x7fdbb40fc260 msgr2=0x7fdbb40fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 -- 172.21.15.39:0/2826865156 shutdown_connections 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 -- 172.21.15.39:0/2826865156 wait complete. 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.019+0000 7fdbba793700 1 Processor -- start 2024-02-19T19:03:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbba793700 1 -- start start 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbba793700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbb4100ca0 0x7fdbb41964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbba793700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbba793700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb4197040 con 0x7fdbb4101ea0 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbba793700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb4197180 con 0x7fdbb4100ca0 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41880/0 (socket says 172.21.15.39:41880) 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 -- 172.21.15.39:0/3580312728 learned_addr learned my addr 172.21.15.39:0/3580312728 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb3fff700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbb4100ca0 0x7fdbb41964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 -- 172.21.15.39:0/3580312728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbb4100ca0 msgr2=0x7fdbb41964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbb4100ca0 0x7fdbb41964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 -- 172.21.15.39:0/3580312728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba80095f0 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.020+0000 7fdbb37fe700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4196a20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdba8006010 tx=0x7fdba800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.021+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdba801d070 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.021+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb419bbd0 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.021+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdba800f460 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.021+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdba8021640 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.021+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb419c0c0 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.022+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb4109800 con 0x7fdbb4101ea0 2024-02-19T19:03:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.022+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdba800fab0 con 0x7fdbb4101ea0 2024-02-19T19:03:58.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.022+0000 7fdbb17fa700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.023+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdba808cad0 con 0x7fdbb4101ea0 2024-02-19T19:03:58.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.023+0000 7fdbb3fff700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.023+0000 7fdbb3fff700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4101d00 tx=0x7fdba400b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:58.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.027+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdba805ace0 con 0x7fdbb4101ea0 2024-02-19T19:03:58.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:57 smithi039 ceph-mon[28988]: pgmap v242: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:58.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.243+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbb41067f0 con 0x7fdb9c06c4e0 2024-02-19T19:03:58.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.244+0000 7fdbb17fa700 1 -- 172.21.15.39:0/3580312728 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fdbb41067f0 con 0x7fdb9c06c4e0 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb9c06c4e0 msgr2=0x7fdb9c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4101d00 tx=0x7fdba400b410 comp rx=0 tx=0).stop 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 msgr2=0x7fdbb4196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4196a20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdba8006010 tx=0x7fdba800bd70 comp rx=0 tx=0).stop 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 shutdown_connections 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbb4100ca0 0x7fdbb41964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 --2- 172.21.15.39:0/3580312728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdbb4101ea0 0x7fdbb4196a20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.246+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 >> 172.21.15.39:0/3580312728 conn(0x7fdbb40fc260 msgr2=0x7fdbb41050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.247+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 shutdown_connections 2024-02-19T19:03:58.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.247+0000 7fdbba793700 1 -- 172.21.15.39:0/3580312728 wait complete. 2024-02-19T19:03:58.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:57 smithi086 ceph-mon[37629]: pgmap v242: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:03:58.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.352+0000 7f359ebcf700 1 -- 172.21.15.39:0/1736888926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 msgr2=0x7f3598101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.352+0000 7f359ebcf700 1 --2- 172.21.15.39:0/1736888926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f3598101560 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f358c009910 tx=0x7f358c009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.352+0000 7f359ebcf700 1 -- 172.21.15.39:0/1736888926 shutdown_connections 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.352+0000 7f359ebcf700 1 --2- 172.21.15.39:0/1736888926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 0x7f35981002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.352+0000 7f359ebcf700 1 --2- 172.21.15.39:0/1736888926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f3598101560 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.352+0000 7f359ebcf700 1 -- 172.21.15.39:0/1736888926 >> 172.21.15.39:0/1736888926 conn(0x7f35980fa8b0 msgr2=0x7f35980fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 -- 172.21.15.39:0/1736888926 shutdown_connections 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 -- 172.21.15.39:0/1736888926 wait complete. 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 Processor -- start 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 -- start start 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 0x7f35980ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f359819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359819b820 con 0x7f3598101140 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359ebcf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359819b960 con 0x7f35980ffe80 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f3597fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f359819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f359c96b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 0x7f35980ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f3597fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f359819b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41886/0 (socket says 172.21.15.39:41886) 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.353+0000 7f3597fff700 1 -- 172.21.15.39:0/602152823 learned_addr learned my addr 172.21.15.39:0/602152823 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f359c96b700 1 -- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 msgr2=0x7f359819b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f359c96b700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f359819b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f359c96b700 1 -- 172.21.15.39:0/602152823 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0095f0 con 0x7f35980ffe80 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f3597fff700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f359819b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f359c96b700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 0x7f35980ff8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f358c006010 tx=0x7f358c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358c01d070 con 0x7f35980ffe80 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359819bbe0 con 0x7f35980ffe80 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f358c005dc0 con 0x7f35980ffe80 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358c017400 con 0x7f35980ffe80 2024-02-19T19:03:58.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.354+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359819c0d0 con 0x7f35980ffe80 2024-02-19T19:03:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.355+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3598108720 con 0x7f35980ffe80 2024-02-19T19:03:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.356+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f358c003810 con 0x7f35980ffe80 2024-02-19T19:03:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.356+0000 7f3595ffb700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f358806c4e0 0x7f358806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.356+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f358c08cc10 con 0x7f35980ffe80 2024-02-19T19:03:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.356+0000 7f3597fff700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f358806c4e0 0x7f358806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.357+0000 7f3597fff700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f358806c4e0 0x7f358806e9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35980ffe50 tx=0x7f3584009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:58.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.361+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f358c05ae20 con 0x7f35980ffe80 2024-02-19T19:03:58.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.572+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f35981057e0 con 0x7f358806c4e0 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.582+0000 7f3595ffb700 1 -- 172.21.15.39:0/602152823 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f35981057e0 con 0x7f358806c4e0 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (6m) 88s ago 7m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (7m) 88s ago 7m 7424k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (6m) 93s ago 6m 7382k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (7m) 88s ago 7m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (6m) 93s ago 6m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (5m) 88s ago 7m 87.1M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (109s) 88s ago 108s 14.7M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (116s) 88s ago 115s 11.5M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:03:58.602 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (113s) 93s ago 112s 18.6M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (106s) 93s ago 105s 13.3M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:9283,8765,8443 running (9m) 88s ago 9m 492M - 18.2.1 94eadd990e0a f3c738bbd2c8 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (6m) 93s ago 6m 443M - 18.2.1 94eadd990e0a 1b78d388267f 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (9m) 88s ago 9m 45.8M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (6m) 93s ago 6m 35.5M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (7m) 88s ago 7m 14.5M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (6m) 93s ago 6m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (5m) 88s ago 5m 50.7M 3946M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (5m) 88s ago 5m 48.2M 3946M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (4m) 88s ago 4m 46.7M 3946M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (4m) 93s ago 4m 46.2M 4970M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (3m) 93s ago 3m 43.8M 4970M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (3m) 93s ago 3m 46.0M 4970M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:03:58.603 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (5m) 88s ago 7m 55.5M - 2.43.0 a07b618ecd1d 54ef95da7db1 2024-02-19T19:03:58.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f358806c4e0 msgr2=0x7f358806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f358806c4e0 0x7f358806e9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35980ffe50 tx=0x7f3584009260 comp rx=0 tx=0).stop 2024-02-19T19:03:58.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 msgr2=0x7f35980ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 0x7f35980ff8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f358c006010 tx=0x7f358c00f690 comp rx=0 tx=0).stop 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 shutdown_connections 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f358806c4e0 0x7f358806e9a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35980ffe80 0x7f35980ff8d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 --2- 172.21.15.39:0/602152823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3598101140 0x7f359819b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 >> 172.21.15.39:0/602152823 conn(0x7f35980fa8b0 msgr2=0x7f35981040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 shutdown_connections 2024-02-19T19:03:58.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.585+0000 7f359ebcf700 1 -- 172.21.15.39:0/602152823 wait complete. 2024-02-19T19:03:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 -- 172.21.15.39:0/3512969772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 msgr2=0x7fd8b00fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/3512969772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b00fffc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000b0a0 tx=0x7fd8a000b3b0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 -- 172.21.15.39:0/3512969772 shutdown_connections 2024-02-19T19:03:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/3512969772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 0x7fd8b0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/3512969772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b00fffc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 -- 172.21.15.39:0/3512969772 >> 172.21.15.39:0/3512969772 conn(0x7fd8b00fb120 msgr2=0x7fd8b00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 -- 172.21.15.39:0/3512969772 shutdown_connections 2024-02-19T19:03:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 -- 172.21.15.39:0/3512969772 wait complete. 2024-02-19T19:03:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.690+0000 7fd8b85a5700 1 Processor -- start 2024-02-19T19:03:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b85a5700 1 -- start start 2024-02-19T19:03:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b85a5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b85a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 0x7fd8b0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b85a5700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0197030 con 0x7fd8b00ffba0 2024-02-19T19:03:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b85a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0197170 con 0x7fd8b0100da0 2024-02-19T19:03:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b01964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41900/0 (socket says 172.21.15.39:41900) 2024-02-19T19:03:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 -- 172.21.15.39:0/882725574 learned_addr learned my addr 172.21.15.39:0/882725574 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b5b40700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 0x7fd8b0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 -- 172.21.15.39:0/882725574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 msgr2=0x7fd8b0196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 0x7fd8b0196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 -- 172.21.15.39:0/882725574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0009d00 con 0x7fd8b00ffba0 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b5b40700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 0x7fd8b0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b6341700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b01964d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00077b0 tx=0x7fd8a0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a000e040 con 0x7fd8b00ffba0 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8a000ce60 con 0x7fd8b00ffba0 2024-02-19T19:03:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a001db10 con 0x7fd8b00ffba0 2024-02-19T19:03:58.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.691+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b019bbc0 con 0x7fd8b00ffba0 2024-02-19T19:03:58.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.692+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b019c0b0 con 0x7fd8b00ffba0 2024-02-19T19:03:58.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.692+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b0108700 con 0x7fd8b00ffba0 2024-02-19T19:03:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.693+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd8a001dc90 con 0x7fd8b00ffba0 2024-02-19T19:03:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.693+0000 7fd8af7fe700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd89806c4e0 0x7fd89806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.694+0000 7fd8b5b40700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd89806c4e0 0x7fd89806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.694+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd8a008d330 con 0x7fd8b00ffba0 2024-02-19T19:03:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.694+0000 7fd8b5b40700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd89806c4e0 0x7fd89806e9a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4005fd0 tx=0x7fd8a4005dc0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.698+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd8a005b540 con 0x7fd8b00ffba0 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.982+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd8b019c660 con 0x7fd8b00ffba0 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.983+0000 7fd8af7fe700 1 -- 172.21.15.39:0/882725574 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fd8a005b0d0 con 0x7fd8b00ffba0 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:59.002 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:03:59.003 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:03:59.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd89806c4e0 msgr2=0x7fd89806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd89806c4e0 0x7fd89806e9a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4005fd0 tx=0x7fd8a4005dc0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 msgr2=0x7fd8b01964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b01964d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00077b0 tx=0x7fd8a0003ce0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 shutdown_connections 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd89806c4e0 0x7fd89806e9a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8b00ffba0 0x7fd8b01964d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 --2- 172.21.15.39:0/882725574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0100da0 0x7fd8b0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 >> 172.21.15.39:0/882725574 conn(0x7fd8b00fb120 msgr2=0x7fd8b0068e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 shutdown_connections 2024-02-19T19:03:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:58.985+0000 7fd8b85a5700 1 -- 172.21.15.39:0/882725574 wait complete. 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 -- 172.21.15.39:0/3762741423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 msgr2=0x7fb5dc103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 --2- 172.21.15.39:0/3762741423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc103bf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009910 tx=0x7fb5cc009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 -- 172.21.15.39:0/3762741423 shutdown_connections 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 --2- 172.21.15.39:0/3762741423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5dc104130 0x7fb5dc106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 --2- 172.21.15.39:0/3762741423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc103bf0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 -- 172.21.15.39:0/3762741423 >> 172.21.15.39:0/3762741423 conn(0x7fb5dc0fb190 msgr2=0x7fb5dc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 -- 172.21.15.39:0/3762741423 shutdown_connections 2024-02-19T19:03:59.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.094+0000 7fb5e4616700 1 -- 172.21.15.39:0/3762741423 wait complete. 2024-02-19T19:03:59.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e4616700 1 Processor -- start 2024-02-19T19:03:59.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e4616700 1 -- start start 2024-02-19T19:03:59.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e4616700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e4616700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5dc104130 0x7fb5dc1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e4616700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc196fe0 con 0x7fb5dc101800 2024-02-19T19:03:59.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e4616700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc197120 con 0x7fb5dc104130 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41916/0 (socket says 172.21.15.39:41916) 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 -- 172.21.15.39:0/641397078 learned_addr learned my addr 172.21.15.39:0/641397078 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e1bb1700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5dc104130 0x7fb5dc1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 -- 172.21.15.39:0/641397078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5dc104130 msgr2=0x7fb5dc1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5dc104130 0x7fb5dc1969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 -- 172.21.15.39:0/641397078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0095f0 con 0x7fb5dc101800 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.095+0000 7fb5e23b2700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc196480 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc00b5c0 tx=0x7fb5cc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.096+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5cc01d070 con 0x7fb5dc101800 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.096+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb5cc00f460 con 0x7fb5dc101800 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.096+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5cc021640 con 0x7fb5dc101800 2024-02-19T19:03:59.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.096+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc19bb70 con 0x7fb5dc101800 2024-02-19T19:03:59.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.096+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc19c060 con 0x7fb5dc101800 2024-02-19T19:03:59.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.097+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc190610 con 0x7fb5dc101800 2024-02-19T19:03:59.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.097+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb5cc00fa60 con 0x7fb5dc101800 2024-02-19T19:03:59.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.098+0000 7fb5db7fe700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb5c406c4e0 0x7fb5c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.098+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb5cc08c9e0 con 0x7fb5dc101800 2024-02-19T19:03:59.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.098+0000 7fb5e1bb1700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb5c406c4e0 0x7fb5c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.098+0000 7fb5e1bb1700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb5c406c4e0 0x7fb5c406e9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0005950 tx=0x7fb5d000b410 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:59.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.102+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb5cc05ac70 con 0x7fb5dc101800 2024-02-19T19:03:59.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.344+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb5dc02cc70 con 0x7fb5dc101800 2024-02-19T19:03:59.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.344+0000 7fb5db7fe700 1 -- 172.21.15.39:0/641397078 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1847 (secure 0 0 0) 0x7fb5cc026750 con 0x7fb5dc101800 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:03:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:03:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:59.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb5c406c4e0 msgr2=0x7fb5c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb5c406c4e0 0x7fb5c406e9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0005950 tx=0x7fb5d000b410 comp rx=0 tx=0).stop 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 msgr2=0x7fb5dc196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc196480 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc00b5c0 tx=0x7fb5cc00bd70 comp rx=0 tx=0).stop 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 shutdown_connections 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fb5c406c4e0 0x7fb5c406e9a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5dc101800 0x7fb5dc196480 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 --2- 172.21.15.39:0/641397078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5dc104130 0x7fb5dc1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 >> 172.21.15.39:0/641397078 conn(0x7fb5dc0fb190 msgr2=0x7fb5dc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 shutdown_connections 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.347+0000 7fb5e4616700 1 -- 172.21.15.39:0/641397078 wait complete. 2024-02-19T19:03:59.368 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:03:59.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 -- 172.21.15.39:0/3225107342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc100e10 msgr2=0x7f3ffc101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 --2- 172.21.15.39:0/3225107342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc100e10 0x7f3ffc101270 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009910 tx=0x7f3ff0009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:59.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 -- 172.21.15.39:0/3225107342 shutdown_connections 2024-02-19T19:03:59.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 --2- 172.21.15.39:0/3225107342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc100e10 0x7f3ffc101270 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 --2- 172.21.15.39:0/3225107342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 -- 172.21.15.39:0/3225107342 >> 172.21.15.39:0/3225107342 conn(0x7f3ffc0fb190 msgr2=0x7f3ffc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:59.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 -- 172.21.15.39:0/3225107342 shutdown_connections 2024-02-19T19:03:59.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 -- 172.21.15.39:0/3225107342 wait complete. 2024-02-19T19:03:59.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.454+0000 7f4001e21700 1 Processor -- start 2024-02-19T19:03:59.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f4001e21700 1 -- start start 2024-02-19T19:03:59.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f4001e21700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f4001e21700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ffc100e10 0x7f3ffc1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f4001e21700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc196fe0 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f4001e21700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc197120 con 0x7f3ffc100e10 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41936/0 (socket says 172.21.15.39:41936) 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 -- 172.21.15.39:0/2144608718 learned_addr learned my addr 172.21.15.39:0/2144608718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffaffd700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ffc100e10 0x7f3ffc1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 -- 172.21.15.39:0/2144608718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ffc100e10 msgr2=0x7f3ffc1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ffc100e10 0x7f3ffc1969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 -- 172.21.15.39:0/2144608718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff00095f0 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.455+0000 7f3ffb7fe700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc196480 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00e8f0 tx=0x7f3fec00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.456+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fec009790 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.456+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3fec00c810 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.456+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc19bbd0 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.456+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fec003d20 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.456+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc19c120 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.457+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ffc108700 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.457+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3fec0041f0 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.458+0000 7f3ff8ff9700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f3fe406c4e0 0x7f3fe406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.458+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3fec014070 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.458+0000 7f3ffaffd700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f3fe406c4e0 0x7f3fe406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.458+0000 7f3ffaffd700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f3fe406c4e0 0x7f3fe406e9a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009fd0 tx=0x7f3ff0005fd0 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:59.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.462+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3fec059d90 con 0x7f3ffc0ffc10 2024-02-19T19:03:59.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.679+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ffc068c90 con 0x7f3fe406c4e0 2024-02-19T19:03:59.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.681+0000 7f3ff8ff9700 1 -- 172.21.15.39:0/2144608718 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f3ffc068c90 con 0x7f3fe406c4e0 2024-02-19T19:03:59.700 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:03:59.700 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:03:59.700 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:03:59.701 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:03:59.701 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [], 2024-02-19T19:03:59.701 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "0/23 daemons upgraded", 2024-02-19T19:03:59.701 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 image on host smithi086", 2024-02-19T19:03:59.701 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:03:59.701 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:03:59.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f3fe406c4e0 msgr2=0x7f3fe406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f3fe406c4e0 0x7f3fe406e9a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009fd0 tx=0x7f3ff0005fd0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 msgr2=0x7f3ffc196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc196480 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00e8f0 tx=0x7f3fec00ec00 comp rx=0 tx=0).stop 2024-02-19T19:03:59.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 shutdown_connections 2024-02-19T19:03:59.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f3fe406c4e0 0x7f3fe406e9a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ffc0ffc10 0x7f3ffc196480 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 --2- 172.21.15.39:0/2144608718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ffc100e10 0x7f3ffc1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 >> 172.21.15.39:0/2144608718 conn(0x7f3ffc0fb190 msgr2=0x7f3ffc068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:59.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 shutdown_connections 2024-02-19T19:03:59.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.683+0000 7f4001e21700 1 -- 172.21.15.39:0/2144608718 wait complete. 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 -- 172.21.15.39:0/4143837776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf80ffbc0 msgr2=0x7facf80fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 --2- 172.21.15.39:0/4143837776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf80ffbc0 0x7facf80fffe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7facf4009910 tx=0x7facf4009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 -- 172.21.15.39:0/4143837776 shutdown_connections 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 --2- 172.21.15.39:0/4143837776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf8100dc0 0x7facf8101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 --2- 172.21.15.39:0/4143837776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf80ffbc0 0x7facf80fffe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 -- 172.21.15.39:0/4143837776 >> 172.21.15.39:0/4143837776 conn(0x7facf80fb160 msgr2=0x7facf80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 -- 172.21.15.39:0/4143837776 shutdown_connections 2024-02-19T19:03:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 -- 172.21.15.39:0/4143837776 wait complete. 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.789+0000 7facff9a1700 1 Processor -- start 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facff9a1700 1 -- start start 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facff9a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf80ffbc0 0x7facf8196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facff9a1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 0x7facf8196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facff9a1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf8197360 con 0x7facf8100dc0 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facff9a1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf81974a0 con 0x7facf80ffbc0 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 0x7facf8196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 0x7facf8196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41968/0 (socket says 172.21.15.39:41968) 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 -- 172.21.15.39:0/3894649178 learned_addr learned my addr 172.21.15.39:0/3894649178 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:03:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfd73d700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf80ffbc0 0x7facf8196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 -- 172.21.15.39:0/3894649178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf80ffbc0 msgr2=0x7facf8196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf80ffbc0 0x7facf8196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 -- 172.21.15.39:0/3894649178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40095f0 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfd73d700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf80ffbc0 0x7facf8196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.790+0000 7facfcf3c700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 0x7facf8196d40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7face800e920 tx=0x7face800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.791+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7face800c9d0 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.791+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7face8010430 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.791+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf819bf50 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.791+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7face8018650 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.791+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf819c470 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.792+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf8068af0 con 0x7facf8100dc0 2024-02-19T19:03:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.792+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7face80105a0 con 0x7facf8100dc0 2024-02-19T19:03:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.792+0000 7facee7fc700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7face406c4e0 0x7face406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.793+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7face8014070 con 0x7facf8100dc0 2024-02-19T19:03:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.793+0000 7facfd73d700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7face406c4e0 0x7face406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.793+0000 7facfd73d700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7face406c4e0 0x7face406e9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facf400b6f0 tx=0x7facf4005940 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:03:59.797+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7face80520e0 con 0x7facf8100dc0 2024-02-19T19:03:59.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:59 smithi039 ceph-mon[28988]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:59.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:59 smithi039 ceph-mon[28988]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:59.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:59 smithi039 ceph-mon[28988]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:59.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:59 smithi039 ceph-mon[28988]: pgmap v243: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:03:59.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:59 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/882725574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:59.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:03:59 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/641397078' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.083+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7facf819c6e0 con 0x7facf8100dc0 2024-02-19T19:04:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.084+0000 7facee7fc700 1 -- 172.21.15.39:0/3894649178 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7face8059d90 con 0x7facf8100dc0 2024-02-19T19:04:00.103 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:00.103 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:00.103 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:04:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7face406c4e0 msgr2=0x7face406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7face406c4e0 0x7face406e9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7facf400b6f0 tx=0x7facf4005940 comp rx=0 tx=0).stop 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 msgr2=0x7facf8196d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 0x7facf8196d40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7face800e920 tx=0x7face800ec30 comp rx=0 tx=0).stop 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 shutdown_connections 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7face406c4e0 0x7face406e9a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facf80ffbc0 0x7facf8196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 --2- 172.21.15.39:0/3894649178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facf8100dc0 0x7facf8196d40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 >> 172.21.15.39:0/3894649178 conn(0x7facf80fb160 msgr2=0x7facf8068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 shutdown_connections 2024-02-19T19:04:00.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:00.086+0000 7facff9a1700 1 -- 172.21.15.39:0/3894649178 wait complete. 2024-02-19T19:04:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:59 smithi086 ceph-mon[37629]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:59 smithi086 ceph-mon[37629]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:59 smithi086 ceph-mon[37629]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:59 smithi086 ceph-mon[37629]: pgmap v243: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:59 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/882725574' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:00.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:03:59 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/641397078' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:01.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:00 smithi039 ceph-mon[28988]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:01.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:00 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3894649178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:01.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:00 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:01.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:00 smithi086 ceph-mon[37629]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:01.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:00 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3894649178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:01.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:00 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:02.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:01 smithi039 ceph-mon[28988]: pgmap v244: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:02.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:01 smithi086 ceph-mon[37629]: pgmap v244: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:04.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:03 smithi039 ceph-mon[28988]: pgmap v245: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:04.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:03 smithi086 ceph-mon[37629]: pgmap v245: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:05.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:05 smithi086 ceph-mon[37629]: pgmap v246: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:05.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:05 smithi039 ceph-mon[28988]: pgmap v246: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:07.996 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:07 smithi039 ceph-mon[28988]: pgmap v247: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:08.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:07 smithi086 ceph-mon[37629]: pgmap v247: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:10.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:09 smithi039 ceph-mon[28988]: pgmap v248: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:10.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:09 smithi086 ceph-mon[37629]: pgmap v248: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:12.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:11 smithi039 ceph-mon[28988]: pgmap v249: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:12.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:11 smithi086 ceph-mon[37629]: pgmap v249: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 852 B/s rd, 1 op/s 2024-02-19T19:04:14.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:13 smithi039 ceph-mon[28988]: pgmap v250: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:14.283 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:13 smithi086 ceph-mon[37629]: pgmap v250: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:15.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:15 smithi086 ceph-mon[37629]: pgmap v251: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:15.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:15 smithi039 ceph-mon[28988]: pgmap v251: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:16.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:16 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:16.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:16 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:17.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:17 smithi039 ceph-mon[28988]: pgmap v252: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:17.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:17 smithi086 ceph-mon[37629]: pgmap v252: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:20.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:19 smithi039 ceph-mon[28988]: pgmap v253: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:20.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:19 smithi086 ceph-mon[37629]: pgmap v253: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:22.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:21 smithi039 ceph-mon[28988]: pgmap v254: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:22.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:21 smithi086 ceph-mon[37629]: pgmap v254: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:24.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:23 smithi039 ceph-mon[28988]: pgmap v255: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:24.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:23 smithi086 ceph-mon[37629]: pgmap v255: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:25.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:25 smithi086 ceph-mon[37629]: pgmap v256: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:25.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:25 smithi039 ceph-mon[28988]: pgmap v256: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:28.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:27 smithi039 ceph-mon[28988]: pgmap v257: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:28.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:27 smithi086 ceph-mon[37629]: pgmap v257: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 -- 172.21.15.39:0/1921656733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8714102280 msgr2=0x7f87141026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 --2- 172.21.15.39:0/1921656733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8714102280 0x7f87141026a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8704009910 tx=0x7f8704009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 -- 172.21.15.39:0/1921656733 shutdown_connections 2024-02-19T19:04:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 --2- 172.21.15.39:0/1921656733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87140fed00 0x7f87140ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 --2- 172.21.15.39:0/1921656733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8714102280 0x7f87141026a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 -- 172.21.15.39:0/1921656733 >> 172.21.15.39:0/1921656733 conn(0x7f87140fa8c0 msgr2=0x7f87140fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:30.225 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:29 smithi039 ceph-mon[28988]: pgmap v258: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 -- 172.21.15.39:0/1921656733 shutdown_connections 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.204+0000 7f871a511700 1 -- 172.21.15.39:0/1921656733 wait complete. 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.205+0000 7f871a511700 1 Processor -- start 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.205+0000 7f871a511700 1 -- start start 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.205+0000 7f871a511700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 0x7f8714196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.205+0000 7f871a511700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714102280 0x7f8714196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.205+0000 7f871a511700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87141970b0 con 0x7f87140fed00 2024-02-19T19:04:30.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.205+0000 7f871a511700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87141971f0 con 0x7f8714102280 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 0x7f8714196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 0x7f8714196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34996/0 (socket says 172.21.15.39:34996) 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 -- 172.21.15.39:0/3478717308 learned_addr learned my addr 172.21.15.39:0/3478717308 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f87137fe700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714102280 0x7f8714196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 -- 172.21.15.39:0/3478717308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714102280 msgr2=0x7f8714196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714102280 0x7f8714196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 -- 172.21.15.39:0/3478717308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040095f0 con 0x7f87140fed00 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f8713fff700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 0x7f8714196550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8704000c00 tx=0x7f870400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f870401d070 con 0x7f87140fed00 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8704005dc0 con 0x7f87140fed00 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871419bc40 con 0x7f87140fed00 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8704022470 con 0x7f87140fed00 2024-02-19T19:04:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.206+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871419c130 con 0x7f87140fed00 2024-02-19T19:04:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.207+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8714109810 con 0x7f87140fed00 2024-02-19T19:04:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.208+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8704003810 con 0x7f87140fed00 2024-02-19T19:04:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.208+0000 7f87117fa700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f86fc06c4e0 0x7f86fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.209+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f870408cd30 con 0x7f87140fed00 2024-02-19T19:04:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.209+0000 7f87137fe700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f86fc06c4e0 0x7f86fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.209+0000 7f87137fe700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f86fc06c4e0 0x7f86fc06e9a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8708009c60 tx=0x7f8708009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:30.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.213+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f870405af40 con 0x7f87140fed00 2024-02-19T19:04:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:29 smithi086 ceph-mon[37629]: pgmap v258: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:30.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.429+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8714106970 con 0x7f86fc06c4e0 2024-02-19T19:04:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.430+0000 7f87117fa700 1 -- 172.21.15.39:0/3478717308 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f8714106970 con 0x7f86fc06c4e0 2024-02-19T19:04:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f86fc06c4e0 msgr2=0x7f86fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f86fc06c4e0 0x7f86fc06e9a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8708009c60 tx=0x7f8708009260 comp rx=0 tx=0).stop 2024-02-19T19:04:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 msgr2=0x7f8714196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 0x7f8714196550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8704000c00 tx=0x7f870400f690 comp rx=0 tx=0).stop 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 shutdown_connections 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f86fc06c4e0 0x7f86fc06e9a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140fed00 0x7f8714196550 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 --2- 172.21.15.39:0/3478717308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714102280 0x7f8714196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 >> 172.21.15.39:0/3478717308 conn(0x7f87140fa8c0 msgr2=0x7f8714105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 shutdown_connections 2024-02-19T19:04:30.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.432+0000 7f871a511700 1 -- 172.21.15.39:0/3478717308 wait complete. 2024-02-19T19:04:30.462 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:04:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 -- 172.21.15.39:0/1525438271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 msgr2=0x7f43b4101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/1525438271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4101210 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f43a8009910 tx=0x7f43a8009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 -- 172.21.15.39:0/1525438271 shutdown_connections 2024-02-19T19:04:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/1525438271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4101210 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/1525438271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43b40ffbb0 0x7f43b40fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 -- 172.21.15.39:0/1525438271 >> 172.21.15.39:0/1525438271 conn(0x7f43b40fb150 msgr2=0x7f43b40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 -- 172.21.15.39:0/1525438271 shutdown_connections 2024-02-19T19:04:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.538+0000 7f43b8c5b700 1 -- 172.21.15.39:0/1525438271 wait complete. 2024-02-19T19:04:30.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b8c5b700 1 Processor -- start 2024-02-19T19:04:30.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b8c5b700 1 -- start start 2024-02-19T19:04:30.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b8c5b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43b40ffbb0 0x7f43b4103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b8c5b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4102190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b8c5b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41026d0 con 0x7f43b4100db0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b8c5b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4102810 con 0x7f43b40ffbb0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4102190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4102190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35018/0 (socket says 172.21.15.39:35018) 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b1d9b700 1 -- 172.21.15.39:0/3584396963 learned_addr learned my addr 172.21.15.39:0/3584396963 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b259c700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43b40ffbb0 0x7f43b4103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b1d9b700 1 -- 172.21.15.39:0/3584396963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43b40ffbb0 msgr2=0x7f43b4103b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b1d9b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43b40ffbb0 0x7f43b4103b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.539+0000 7f43b1d9b700 1 -- 172.21.15.39:0/3584396963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a80095f0 con 0x7f43b4100db0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.540+0000 7f43b1d9b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4102190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f43a8000c00 tx=0x7f43a800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.540+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a801d070 con 0x7f43b4100db0 2024-02-19T19:04:30.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.540+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f43a8005dc0 con 0x7f43b4100db0 2024-02-19T19:04:30.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.540+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b4102a90 con 0x7f43b4100db0 2024-02-19T19:04:30.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.540+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a80226a0 con 0x7f43b4100db0 2024-02-19T19:04:30.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.540+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b4102f80 con 0x7f43b4100db0 2024-02-19T19:04:30.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.541+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b4108710 con 0x7f43b4100db0 2024-02-19T19:04:30.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.541+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f43a80038f0 con 0x7f43b4100db0 2024-02-19T19:04:30.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.541+0000 7f43a37fe700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f439c06c4e0 0x7f439c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.542+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f43a808cd60 con 0x7f43b4100db0 2024-02-19T19:04:30.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.542+0000 7f43b259c700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f439c06c4e0 0x7f439c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.542+0000 7f43b259c700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f439c06c4e0 0x7f439c06e9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43b4100c10 tx=0x7f43a4009260 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:30.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.546+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f43a805af70 con 0x7f43b4100db0 2024-02-19T19:04:30.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.765+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43b419e970 con 0x7f439c06c4e0 2024-02-19T19:04:30.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.766+0000 7f43a37fe700 1 -- 172.21.15.39:0/3584396963 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f43b419e970 con 0x7f439c06c4e0 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f439c06c4e0 msgr2=0x7f439c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f439c06c4e0 0x7f439c06e9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43b4100c10 tx=0x7f43a4009260 comp rx=0 tx=0).stop 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 msgr2=0x7f43b4102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4102190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f43a8000c00 tx=0x7f43a800f690 comp rx=0 tx=0).stop 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 shutdown_connections 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f439c06c4e0 0x7f439c06e9a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43b40ffbb0 0x7f43b4103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 --2- 172.21.15.39:0/3584396963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43b4100db0 0x7f43b4102190 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 >> 172.21.15.39:0/3584396963 conn(0x7f43b40fb150 msgr2=0x7f43b4068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 shutdown_connections 2024-02-19T19:04:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.768+0000 7f43b8c5b700 1 -- 172.21.15.39:0/3584396963 wait complete. 2024-02-19T19:04:30.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 -- 172.21.15.39:0/829640788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b8100e10 msgr2=0x7f83b8101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 --2- 172.21.15.39:0/829640788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b8100e10 0x7f83b8101270 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009910 tx=0x7f83ac009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:30.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 -- 172.21.15.39:0/829640788 shutdown_connections 2024-02-19T19:04:30.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 --2- 172.21.15.39:0/829640788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b8100e10 0x7f83b8101270 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 --2- 172.21.15.39:0/829640788 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b80ffc10 0x7f83b8100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 -- 172.21.15.39:0/829640788 >> 172.21.15.39:0/829640788 conn(0x7f83b80fb190 msgr2=0x7f83b80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 -- 172.21.15.39:0/829640788 shutdown_connections 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 -- 172.21.15.39:0/829640788 wait complete. 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.877+0000 7f83bdfda700 1 Processor -- start 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83bdfda700 1 -- start start 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83bdfda700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 0x7f83b80690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83bdfda700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b8100e10 0x7f83b819b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83bdfda700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b819b7d0 con 0x7f83b80ffc10 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83bdfda700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b819b910 con 0x7f83b8100e10 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 0x7f83b80690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 0x7f83b80690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35044/0 (socket says 172.21.15.39:35044) 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 -- 172.21.15.39:0/3460074432 learned_addr learned my addr 172.21.15.39:0/3460074432 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:30.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b6ffd700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b8100e10 0x7f83b819b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 -- 172.21.15.39:0/3460074432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b8100e10 msgr2=0x7f83b819b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b8100e10 0x7f83b819b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 -- 172.21.15.39:0/3460074432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0095f0 con 0x7f83b80ffc10 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.878+0000 7f83b77fe700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 0x7f83b80690e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f83a800e8f0 tx=0x7f83a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.879+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a8009790 con 0x7f83b80ffc10 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.879+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f83a80044d0 con 0x7f83b80ffc10 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.879+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b819bbf0 con 0x7f83b80ffc10 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.879+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a8003dc0 con 0x7f83b80ffc10 2024-02-19T19:04:30.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.879+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b819c140 con 0x7f83b80ffc10 2024-02-19T19:04:30.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.880+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8108700 con 0x7f83b80ffc10 2024-02-19T19:04:30.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.880+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f83a800cba0 con 0x7f83b80ffc10 2024-02-19T19:04:30.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.881+0000 7f83b4ff9700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f83a006c4e0 0x7f83a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:30.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.881+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f83a8014070 con 0x7f83b80ffc10 2024-02-19T19:04:30.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.881+0000 7f83b6ffd700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f83a006c4e0 0x7f83a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:30.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.881+0000 7f83b6ffd700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f83a006c4e0 0x7f83a006e9a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f83b8069660 tx=0x7f83ac009f70 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:30.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:30.885+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f83a8059da0 con 0x7f83b80ffc10 2024-02-19T19:04:31.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.092+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b8105760 con 0x7f83a006c4e0 2024-02-19T19:04:31.113 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:30 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:31.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.103+0000 7f83b4ff9700 1 -- 172.21.15.39:0/3460074432 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3456 (secure 0 0 0) 0x7f83b8105760 con 0x7f83a006c4e0 2024-02-19T19:04:31.123 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:04:31.123 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (6m) 2m ago 8m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (8m) 2m ago 8m 7424k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (7m) 2m ago 7m 7382k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (8m) 2m ago 8m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (7m) 2m ago 7m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (6m) 2m ago 7m 87.1M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (2m) 2m ago 2m 14.7M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (2m) 2m ago 2m 11.5M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (2m) 2m ago 2m 18.6M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (2m) 2m ago 2m 13.3M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:9283,8765,8443 running (9m) 2m ago 9m 492M - 18.2.1 94eadd990e0a f3c738bbd2c8 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (7m) 2m ago 7m 443M - 18.2.1 94eadd990e0a 1b78d388267f 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (9m) 2m ago 9m 45.8M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (6m) 2m ago 6m 35.5M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:04:31.124 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (8m) 2m ago 8m 14.5M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (7m) 2m ago 7m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (6m) 2m ago 6m 50.7M 3946M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (5m) 2m ago 5m 48.2M 3946M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (5m) 2m ago 5m 46.7M 3946M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (4m) 2m ago 4m 46.2M 4970M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (4m) 2m ago 4m 43.8M 4970M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (3m) 2m ago 3m 46.0M 4970M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (6m) 2m ago 7m 55.5M - 2.43.0 a07b618ecd1d 54ef95da7db1 2024-02-19T19:04:31.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f83a006c4e0 msgr2=0x7f83a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f83a006c4e0 0x7f83a006e9a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f83b8069660 tx=0x7f83ac009f70 comp rx=0 tx=0).stop 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 msgr2=0x7f83b80690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 0x7f83b80690e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f83a800e8f0 tx=0x7f83a800ecb0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 shutdown_connections 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f83a006c4e0 0x7f83a006e9a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83b80ffc10 0x7f83b80690e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 --2- 172.21.15.39:0/3460074432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b8100e10 0x7f83b819b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 >> 172.21.15.39:0/3460074432 conn(0x7f83b80fb190 msgr2=0x7f83b8104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 shutdown_connections 2024-02-19T19:04:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.105+0000 7f83bdfda700 1 -- 172.21.15.39:0/3460074432 wait complete. 2024-02-19T19:04:31.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.209+0000 7fd3e41b5700 1 -- 172.21.15.39:0/57745496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 msgr2=0x7fd3dc073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.209+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/57745496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc073bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009910 tx=0x7fd3cc009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.209+0000 7fd3e41b5700 1 -- 172.21.15.39:0/57745496 shutdown_connections 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.209+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/57745496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc073bf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/57745496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3dc074d40 0x7fd3dc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 -- 172.21.15.39:0/57745496 >> 172.21.15.39:0/57745496 conn(0x7fd3dc0fc230 msgr2=0x7fd3dc0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 -- 172.21.15.39:0/57745496 shutdown_connections 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 -- 172.21.15.39:0/57745496 wait complete. 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 Processor -- start 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 -- start start 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3dc074d40 0x7fd3dc198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc199200 con 0x7fd3dc073770 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e41b5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc199340 con 0x7fd3dc074d40 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e1f51700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.210+0000 7fd3e1f51700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc1986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35062/0 (socket says 172.21.15.39:35062) 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e1f51700 1 -- 172.21.15.39:0/2253232393 learned_addr learned my addr 172.21.15.39:0/2253232393 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e1750700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3dc074d40 0x7fd3dc198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e1f51700 1 -- 172.21.15.39:0/2253232393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3dc074d40 msgr2=0x7fd3dc198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e1f51700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3dc074d40 0x7fd3dc198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e1f51700 1 -- 172.21.15.39:0/2253232393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc0095f0 con 0x7fd3dc073770 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e1f51700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc1986a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800d8d0 tx=0x7fd3d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3d8009940 con 0x7fd3dc073770 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd3d8010460 con 0x7fd3dc073770 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc19ddf0 con 0x7fd3dc073770 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3d800f5f0 con 0x7fd3dc073770 2024-02-19T19:04:31.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.211+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc19e270 con 0x7fd3dc073770 2024-02-19T19:04:31.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.212+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc10ba00 con 0x7fd3dc073770 2024-02-19T19:04:31.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.213+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd3d80105d0 con 0x7fd3dc073770 2024-02-19T19:04:31.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.213+0000 7fd3d2ffd700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd3c806c4e0 0x7fd3c806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.213+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd3d808b790 con 0x7fd3dc073770 2024-02-19T19:04:31.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.213+0000 7fd3e1750700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd3c806c4e0 0x7fd3c806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.214+0000 7fd3e1750700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd3c806c4e0 0x7fd3c806e9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc005b40 tx=0x7fd3cc005a90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:31.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.218+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd3d8059a20 con 0x7fd3dc073770 2024-02-19T19:04:31.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:30 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:31.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.503+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd3dc19e900 con 0x7fd3dc073770 2024-02-19T19:04:31.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.503+0000 7fd3d2ffd700 1 -- 172.21.15.39:0/2253232393 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fd3d8016240 con 0x7fd3dc073770 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:04:31.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:04:31.525 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:04:31.525 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:04:31.525 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:04:31.525 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd3c806c4e0 msgr2=0x7fd3c806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd3c806c4e0 0x7fd3c806e9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc005b40 tx=0x7fd3cc005a90 comp rx=0 tx=0).stop 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 msgr2=0x7fd3dc1986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc1986a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800d8d0 tx=0x7fd3d800dc90 comp rx=0 tx=0).stop 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 shutdown_connections 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fd3c806c4e0 0x7fd3c806e9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3dc073770 0x7fd3dc1986a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 --2- 172.21.15.39:0/2253232393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3dc074d40 0x7fd3dc198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 >> 172.21.15.39:0/2253232393 conn(0x7fd3dc0fc230 msgr2=0x7fd3dc106b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 shutdown_connections 2024-02-19T19:04:31.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.505+0000 7fd3e41b5700 1 -- 172.21.15.39:0/2253232393 wait complete. 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 -- 172.21.15.39:0/3333080536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4250100dc0 msgr2=0x7f4250101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 --2- 172.21.15.39:0/3333080536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4250100dc0 0x7f4250101220 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f423c009910 tx=0x7f423c009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 -- 172.21.15.39:0/3333080536 shutdown_connections 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 --2- 172.21.15.39:0/3333080536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4250100dc0 0x7f4250101220 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 --2- 172.21.15.39:0/3333080536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42500ffbc0 0x7f42500fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 -- 172.21.15.39:0/3333080536 >> 172.21.15.39:0/3333080536 conn(0x7f42500fb160 msgr2=0x7f42500fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 -- 172.21.15.39:0/3333080536 shutdown_connections 2024-02-19T19:04:31.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.610+0000 7f4256af1700 1 -- 172.21.15.39:0/3333080536 wait complete. 2024-02-19T19:04:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f4256af1700 1 Processor -- start 2024-02-19T19:04:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f4256af1700 1 -- start start 2024-02-19T19:04:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f4256af1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 0x7f4250069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f4256af1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4250100dc0 0x7f425019b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f4256af1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425019bb70 con 0x7f42500ffbc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f4256af1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425019bce0 con 0x7f4250100dc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f425488d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 0x7f4250069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f425488d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 0x7f4250069990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35082/0 (socket says 172.21.15.39:35082) 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f425488d700 1 -- 172.21.15.39:0/1952749549 learned_addr learned my addr 172.21.15.39:0/1952749549 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.611+0000 7f424ffff700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4250100dc0 0x7f425019b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f425488d700 1 -- 172.21.15.39:0/1952749549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4250100dc0 msgr2=0x7f425019b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f425488d700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4250100dc0 0x7f425019b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f425488d700 1 -- 172.21.15.39:0/1952749549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c0095f0 con 0x7f42500ffbc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f425488d700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 0x7f4250069990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f424400b480 tx=0x7f424400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4244011800 con 0x7f42500ffbc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425019bfc0 con 0x7f42500ffbc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4244011e10 con 0x7f42500ffbc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4244010ea0 con 0x7f42500ffbc0 2024-02-19T19:04:31.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.612+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425019c4b0 con 0x7f42500ffbc0 2024-02-19T19:04:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.613+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425018f5d0 con 0x7f42500ffbc0 2024-02-19T19:04:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.614+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4244010380 con 0x7f42500ffbc0 2024-02-19T19:04:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.614+0000 7f424dffb700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f424006c4e0 0x7f424006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.614+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f424408b4a0 con 0x7f42500ffbc0 2024-02-19T19:04:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.614+0000 7f424ffff700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f424006c4e0 0x7f424006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.614+0000 7f424ffff700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f424006c4e0 0x7f424006e9a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4250069ef0 tx=0x7f423c00b540 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:31.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.618+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f42440596c0 con 0x7f42500ffbc0 2024-02-19T19:04:31.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.864+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4250068af0 con 0x7f42500ffbc0 2024-02-19T19:04:31.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.864+0000 7f424dffb700 1 -- 172.21.15.39:0/1952749549 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1847 (secure 0 0 0) 0x7f4244015750 con 0x7f42500ffbc0 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:04:31.885 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:04:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:04:31.887 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:04:31.888 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:31.888 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:31.888 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:04:31.891 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:04:31.891 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:04:31.891 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:04:31.891 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:31.891 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f424006c4e0 msgr2=0x7f424006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f424006c4e0 0x7f424006e9a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4250069ef0 tx=0x7f423c00b540 comp rx=0 tx=0).stop 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 msgr2=0x7f4250069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 0x7f4250069990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f424400b480 tx=0x7f424400b790 comp rx=0 tx=0).stop 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 shutdown_connections 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f424006c4e0 0x7f424006e9a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42500ffbc0 0x7f4250069990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 --2- 172.21.15.39:0/1952749549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4250100dc0 0x7f425019b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 >> 172.21.15.39:0/1952749549 conn(0x7f42500fb160 msgr2=0x7f42501040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:31.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 shutdown_connections 2024-02-19T19:04:31.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.866+0000 7f4256af1700 1 -- 172.21.15.39:0/1952749549 wait complete. 2024-02-19T19:04:31.893 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:04:31.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 -- 172.21.15.39:0/2260597515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81040e0 msgr2=0x7fe3c81064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/2260597515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81040e0 0x7fe3c81064d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009910 tx=0x7fe3bc009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:31.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 -- 172.21.15.39:0/2260597515 shutdown_connections 2024-02-19T19:04:31.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/2260597515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81040e0 0x7fe3c81064d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/2260597515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c81017b0 0x7fe3c8103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 -- 172.21.15.39:0/2260597515 >> 172.21.15.39:0/2260597515 conn(0x7fe3c80fb160 msgr2=0x7fe3c80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:31.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 -- 172.21.15.39:0/2260597515 shutdown_connections 2024-02-19T19:04:31.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 -- 172.21.15.39:0/2260597515 wait complete. 2024-02-19T19:04:31.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.974+0000 7fe3cd00c700 1 Processor -- start 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3cd00c700 1 -- start start 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3cd00c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 0x7fe3c8101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3cd00c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c81040e0 0x7fe3c80ff760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3cd00c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80ffca0 con 0x7fe3c81017b0 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3cd00c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80ffde0 con 0x7fe3c81040e0 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 0x7fe3c8101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 0x7fe3c8101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35102/0 (socket says 172.21.15.39:35102) 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 -- 172.21.15.39:0/779983067 learned_addr learned my addr 172.21.15.39:0/779983067 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c659c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c81040e0 0x7fe3c80ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 -- 172.21.15.39:0/779983067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c81040e0 msgr2=0x7fe3c80ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c81040e0 0x7fe3c80ff760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:31.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 -- 172.21.15.39:0/779983067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0095f0 con 0x7fe3c81017b0 2024-02-19T19:04:31.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.975+0000 7fe3c6d9d700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 0x7fe3c8101110 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800d8d0 tx=0x7fe3b800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:31.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.976+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3b80098a0 con 0x7fe3c81017b0 2024-02-19T19:04:31.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.976+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c81000c0 con 0x7fe3c81017b0 2024-02-19T19:04:31.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.976+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe3b8010460 con 0x7fe3c81017b0 2024-02-19T19:04:31.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.976+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3b800f5f0 con 0x7fe3c81017b0 2024-02-19T19:04:31.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.976+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c819e800 con 0x7fe3c81017b0 2024-02-19T19:04:31.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.977+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c8190630 con 0x7fe3c81017b0 2024-02-19T19:04:31.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.977+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fe3b800f820 con 0x7fe3c81017b0 2024-02-19T19:04:31.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.977+0000 7fe3affff700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3b006c4e0 0x7fe3b006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:31.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.978+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe3b808b770 con 0x7fe3c81017b0 2024-02-19T19:04:31.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.978+0000 7fe3c659c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3b006c4e0 0x7fe3b006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:31.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.978+0000 7fe3c659c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3b006c4e0 0x7fe3b006e9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc006010 tx=0x7fe3bc005a90 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:32.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:31.982+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe3b80599f0 con 0x7fe3c81017b0 2024-02-19T19:04:32.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:31 smithi086 ceph-mon[37629]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:32.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:31 smithi086 ceph-mon[37629]: pgmap v259: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:32.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:31 smithi086 ceph-mon[37629]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:32.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:31 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2253232393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.202+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 --> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3c802ce60 con 0x7fe3b006c4e0 2024-02-19T19:04:32.222 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:31 smithi039 ceph-mon[28988]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:32.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:31 smithi039 ceph-mon[28988]: pgmap v259: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:32.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:31 smithi039 ceph-mon[28988]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:32.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:31 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2253232393' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.203+0000 7fe3affff700 1 -- 172.21.15.39:0/779983067 <== mgr.14219 v2:172.21.15.39:6800/2307886882 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fe3c802ce60 con 0x7fe3b006c4e0 2024-02-19T19:04:32.223 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:04:32.223 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [], 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "0/23 daemons upgraded", 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 image on host smithi086", 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:04:32.224 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:04:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3b006c4e0 msgr2=0x7fe3b006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3b006c4e0 0x7fe3b006e9a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc006010 tx=0x7fe3bc005a90 comp rx=0 tx=0).stop 2024-02-19T19:04:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 msgr2=0x7fe3c8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 0x7fe3c8101110 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800d8d0 tx=0x7fe3b800dbe0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 shutdown_connections 2024-02-19T19:04:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7fe3b006c4e0 0x7fe3b006e9a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c81017b0 0x7fe3c8101110 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 --2- 172.21.15.39:0/779983067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3c81040e0 0x7fe3c80ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 >> 172.21.15.39:0/779983067 conn(0x7fe3c80fb160 msgr2=0x7fe3c80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.205+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 shutdown_connections 2024-02-19T19:04:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.206+0000 7fe3cd00c700 1 -- 172.21.15.39:0/779983067 wait complete. 2024-02-19T19:04:32.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 -- 172.21.15.39:0/2262586116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e4100e10 msgr2=0x7f10e4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/2262586116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e4100e10 0x7f10e4101270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009910 tx=0x7f10d8009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 -- 172.21.15.39:0/2262586116 shutdown_connections 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/2262586116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e4100e10 0x7f10e4101270 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/2262586116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10e40ffc10 0x7f10e4100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 -- 172.21.15.39:0/2262586116 >> 172.21.15.39:0/2262586116 conn(0x7f10e40fb190 msgr2=0x7f10e40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 -- 172.21.15.39:0/2262586116 shutdown_connections 2024-02-19T19:04:32.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.313+0000 7f10ea5e3700 1 -- 172.21.15.39:0/2262586116 wait complete. 2024-02-19T19:04:32.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10ea5e3700 1 Processor -- start 2024-02-19T19:04:32.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10ea5e3700 1 -- start start 2024-02-19T19:04:32.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10ea5e3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 0x7f10e41953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:32.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10ea5e3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10e4100e10 0x7f10e41958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:32.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10ea5e3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e4195f00 con 0x7f10e40ffc10 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10ea5e3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e4196040 con 0x7f10e4100e10 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 0x7f10e41953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 0x7f10e41953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35114/0 (socket says 172.21.15.39:35114) 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10e3fff700 1 -- 172.21.15.39:0/1948568379 learned_addr learned my addr 172.21.15.39:0/1948568379 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.314+0000 7f10e37fe700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10e4100e10 0x7f10e41958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e3fff700 1 -- 172.21.15.39:0/1948568379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10e4100e10 msgr2=0x7f10e41958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e3fff700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10e4100e10 0x7f10e41958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e3fff700 1 -- 172.21.15.39:0/1948568379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d80095f0 con 0x7f10e40ffc10 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e3fff700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 0x7f10e41953a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f10d400d900 tx=0x7f10d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10d40041f0 con 0x7f10e40ffc10 2024-02-19T19:04:32.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f10d4004350 con 0x7f10e40ffc10 2024-02-19T19:04:32.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e419aaf0 con 0x7f10e40ffc10 2024-02-19T19:04:32.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10d4003d20 con 0x7f10e40ffc10 2024-02-19T19:04:32.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.315+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e419b040 con 0x7f10e40ffc10 2024-02-19T19:04:32.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.316+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e4108700 con 0x7f10e40ffc10 2024-02-19T19:04:32.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.316+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f10d4009730 con 0x7f10e40ffc10 2024-02-19T19:04:32.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.317+0000 7f10e17fa700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f10cc06c4e0 0x7f10cc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:32.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.317+0000 7f10e37fe700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f10cc06c4e0 0x7f10cc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:32.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.317+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f10d408b710 con 0x7f10e40ffc10 2024-02-19T19:04:32.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.317+0000 7f10e37fe700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f10cc06c4e0 0x7f10cc06e9a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10d8006010 tx=0x7f10d800b560 comp rx=0 tx=0).ready entity=mgr.14219 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:32.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.321+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f10d4059920 con 0x7f10e40ffc10 2024-02-19T19:04:32.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.602+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f10e419b650 con 0x7f10e40ffc10 2024-02-19T19:04:32.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.603+0000 7f10e17fa700 1 -- 172.21.15.39:0/1948568379 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f10d4003e80 con 0x7f10e40ffc10 2024-02-19T19:04:32.623 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:32.623 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:32.623 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.604+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f10cc06c4e0 msgr2=0x7f10cc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.604+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f10cc06c4e0 0x7f10cc06e9a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10d8006010 tx=0x7f10d800b560 comp rx=0 tx=0).stop 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.604+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 msgr2=0x7f10e41953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.604+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 0x7f10e41953a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f10d400d900 tx=0x7f10d400dc10 comp rx=0 tx=0).stop 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 shutdown_connections 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:6800/2307886882,v1:172.21.15.39:6801/2307886882] conn(0x7f10cc06c4e0 0x7f10cc06e9a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10e40ffc10 0x7f10e41953a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 --2- 172.21.15.39:0/1948568379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10e4100e10 0x7f10e41958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 >> 172.21.15.39:0/1948568379 conn(0x7f10e40fb190 msgr2=0x7f10e4104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 shutdown_connections 2024-02-19T19:04:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:04:32.605+0000 7f10ea5e3700 1 -- 172.21.15.39:0/1948568379 wait complete. 2024-02-19T19:04:32.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:32.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1952749549' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:32.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1948568379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:32.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:32.883 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:04:32.883 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:04:32.883 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:32 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1952749549' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1948568379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:04:33.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:32 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:34.213 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:33 smithi086 ceph-mon[37629]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:34.213 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:33 smithi086 ceph-mon[37629]: pgmap v260: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:34.213 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:33 smithi086 ceph-mon[37629]: Upgrade: Updating mgr.smithi086.huubbu 2024-02-19T19:04:34.213 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:33 smithi086 ceph-mon[37629]: Deploying daemon mgr.smithi086.huubbu on smithi086 2024-02-19T19:04:34.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:33 smithi039 ceph-mon[28988]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:34.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:33 smithi039 ceph-mon[28988]: pgmap v260: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:34.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:33 smithi039 ceph-mon[28988]: Upgrade: Updating mgr.smithi086.huubbu 2024-02-19T19:04:34.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:33 smithi039 ceph-mon[28988]: Deploying daemon mgr.smithi086.huubbu on smithi086 2024-02-19T19:04:35.069 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:35 smithi086 ceph-mon[37629]: pgmap v261: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:35.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:35 smithi039 ceph-mon[28988]: pgmap v261: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:37.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:37 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:37.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:37 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:37.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:37 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:37.895 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:37 smithi086 ceph-mon[37629]: pgmap v262: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:37.929 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:37 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:37.929 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:37 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:37.929 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:37 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:37.929 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:37 smithi039 ceph-mon[28988]: pgmap v262: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:40.126 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:39 smithi086 ceph-mon[37629]: pgmap v263: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:40.126 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:40.126 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:39 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:40.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:39 smithi039 ceph-mon[28988]: pgmap v263: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:40.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:40.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:39 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:41.695 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:41.695 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:41 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:41.695 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:41 smithi086 ceph-mon[37629]: pgmap v264: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:41.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:41.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:41 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:41.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:41 smithi039 ceph-mon[28988]: pgmap v264: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:04:44.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: pgmap v265: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:44.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu restarted 2024-02-19T19:04:44.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:04:44.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:04:44.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:04:44.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:04:44.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:43 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: pgmap v265: 65 pgs: 65 active+clean; 588 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: Standby manager daemon smithi086.huubbu restarted 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:04:44.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:43 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:04:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: mgrmap e20: smithi039.ydmpuh(active, since 7m), standbys: smithi086.huubbu 2024-02-19T19:04:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr fail", "who": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: osdmap e47: 6 total, 6 up, 6 in 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "mgr fail", "who": "smithi039.ydmpuh"}]': finished 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: mgrmap e21: smithi086.huubbu(active, starting, since 0.00830728s) 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:04:45.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:04:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: Manager daemon smithi086.huubbu is now available 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/trash_purge_schedule"}]: dispatch 2024-02-19T19:04:45.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:44 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/trash_purge_schedule"}]: dispatch 2024-02-19T19:04:45.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: mgrmap e20: smithi039.ydmpuh(active, since 7m), standbys: smithi086.huubbu 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr fail", "who": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: osdmap e47: 6 total, 6 up, 6 in 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.14219 172.21.15.39:0/118550760' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "mgr fail", "who": "smithi039.ydmpuh"}]': finished 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: mgrmap e21: smithi086.huubbu(active, starting, since 0.00830728s) 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:04:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:04:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: Manager daemon smithi086.huubbu is now available 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/trash_purge_schedule"}]: dispatch 2024-02-19T19:04:45.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:44 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi086.huubbu/trash_purge_schedule"}]: dispatch 2024-02-19T19:04:46.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: Deploying cephadm binary to smithi086 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: Deploying cephadm binary to smithi039 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: [19/Feb/2024:19:04:44] ENGINE Bus STARTING 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: [19/Feb/2024:19:04:44] ENGINE Serving on http://172.21.15.86:8765 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: [19/Feb/2024:19:04:44] ENGINE Serving on https://172.21.15.86:7150 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: [19/Feb/2024:19:04:44] ENGINE Bus STARTED 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: mgrmap e22: smithi086.huubbu(active, since 1.01266s) 2024-02-19T19:04:46.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:46 smithi039 ceph-mon[28988]: pgmap v3: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: Deploying cephadm binary to smithi086 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: Deploying cephadm binary to smithi039 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: [19/Feb/2024:19:04:44] ENGINE Bus STARTING 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: [19/Feb/2024:19:04:44] ENGINE Serving on http://172.21.15.86:8765 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: [19/Feb/2024:19:04:44] ENGINE Serving on https://172.21.15.86:7150 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: [19/Feb/2024:19:04:44] ENGINE Bus STARTED 2024-02-19T19:04:46.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: mgrmap e22: smithi086.huubbu(active, since 1.01266s) 2024-02-19T19:04:46.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:46 smithi086 ceph-mon[37629]: pgmap v3: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:47.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:47 smithi039 ceph-mon[28988]: pgmap v4: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:47.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:47 smithi039 ceph-mon[28988]: mgrmap e23: smithi086.huubbu(active, since 2s) 2024-02-19T19:04:47.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:47 smithi086 ceph-mon[37629]: pgmap v4: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:47.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:47 smithi086 ceph-mon[37629]: mgrmap e23: smithi086.huubbu(active, since 2s) 2024-02-19T19:04:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:49 smithi039 ceph-mon[28988]: pgmap v5: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:49 smithi039 ceph-mon[28988]: mgrmap e24: smithi086.huubbu(active, since 4s) 2024-02-19T19:04:49.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:49 smithi086 ceph-mon[37629]: pgmap v5: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:49.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:49 smithi086 ceph-mon[37629]: mgrmap e24: smithi086.huubbu(active, since 4s) 2024-02-19T19:04:51.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:51 smithi039 ceph-mon[28988]: pgmap v6: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:51.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:51 smithi086 ceph-mon[37629]: pgmap v6: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:04:52.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:52 smithi039 ceph-mon[28988]: Standby manager daemon smithi039.ydmpuh started 2024-02-19T19:04:52.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:52 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/crt"}]: dispatch 2024-02-19T19:04:52.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:52 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:04:52.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:52 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/key"}]: dispatch 2024-02-19T19:04:52.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:52 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:04:52.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:52 smithi086 ceph-mon[37629]: Standby manager daemon smithi039.ydmpuh started 2024-02-19T19:04:52.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:52 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/crt"}]: dispatch 2024-02-19T19:04:52.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:52 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:04:52.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:52 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/key"}]: dispatch 2024-02-19T19:04:52.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:52 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/1982584572' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:04:53.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:53 smithi039 ceph-mon[28988]: pgmap v7: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-19T19:04:53.461 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:53 smithi039 ceph-mon[28988]: mgrmap e25: smithi086.huubbu(active, since 8s), standbys: smithi039.ydmpuh 2024-02-19T19:04:53.461 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:53 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:04:53.461 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:53 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:53.461 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:53 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:53.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:53 smithi086 ceph-mon[37629]: pgmap v7: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-19T19:04:53.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:53 smithi086 ceph-mon[37629]: mgrmap e25: smithi086.huubbu(active, since 8s), standbys: smithi039.ydmpuh 2024-02-19T19:04:53.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:53 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:04:53.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:53 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:53.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:53 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:54 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:54 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:54 smithi086 ceph-mon[37629]: pgmap v8: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:04:55.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:54 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:54 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:54 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:54 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:54 smithi039 ceph-mon[28988]: pgmap v8: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:04:55.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:54 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:55.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:54 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:56.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi039 to 1216M 2024-02-19T19:04:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:56 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi039 to 1216M 2024-02-19T19:04:56.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:56 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:04:57.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:57 smithi086 ceph-mon[37629]: pgmap v9: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:04:57.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:57 smithi039 ceph-mon[28988]: pgmap v9: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:04:59.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:59 smithi039 ceph-mon[28988]: pgmap v10: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:04:59.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:04:59 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:59.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:59 smithi086 ceph-mon[37629]: pgmap v10: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:04:59.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:04:59 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:01.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:01 smithi039 ceph-mon[28988]: pgmap v11: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:05:01.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:01 smithi086 ceph-mon[37629]: pgmap v11: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 -- 172.21.15.39:0/894018481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad0100da0 msgr2=0x7fcad0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 --2- 172.21.15.39:0/894018481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad0100da0 0x7fcad0101200 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009910 tx=0x7fcab8009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 -- 172.21.15.39:0/894018481 shutdown_connections 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 --2- 172.21.15.39:0/894018481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad0100da0 0x7fcad0101200 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 --2- 172.21.15.39:0/894018481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad00ffba0 0x7fcad00fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 -- 172.21.15.39:0/894018481 >> 172.21.15.39:0/894018481 conn(0x7fcad00fb120 msgr2=0x7fcad00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 -- 172.21.15.39:0/894018481 shutdown_connections 2024-02-19T19:05:02.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.726+0000 7fcad5645700 1 -- 172.21.15.39:0/894018481 wait complete. 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcad5645700 1 Processor -- start 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcad5645700 1 -- start start 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcad5645700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 0x7fcad0196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcad5645700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad0100da0 0x7fcad01969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcad5645700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0196fe0 con 0x7fcad00ffba0 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcad5645700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad0197120 con 0x7fcad0100da0 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcaceffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 0x7fcad0196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcaceffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 0x7fcad0196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36088/0 (socket says 172.21.15.39:36088) 2024-02-19T19:05:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.727+0000 7fcaceffd700 1 -- 172.21.15.39:0/3974320638 learned_addr learned my addr 172.21.15.39:0/3974320638 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcac67fc700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad0100da0 0x7fcad01969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaceffd700 1 -- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad0100da0 msgr2=0x7fcad01969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaceffd700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad0100da0 0x7fcad01969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaceffd700 1 -- 172.21.15.39:0/3974320638 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab80095f0 con 0x7fcad00ffba0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcac67fc700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad0100da0 0x7fcad01969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaceffd700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 0x7fcad0196480 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcac000e8f0 tx=0x7fcac000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcac0009790 con 0x7fcad00ffba0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcac00044d0 con 0x7fcad00ffba0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad019bbd0 con 0x7fcad00ffba0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcac0003da0 con 0x7fcad00ffba0 2024-02-19T19:05:02.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.728+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad019c120 con 0x7fcad00ffba0 2024-02-19T19:05:02.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.729+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad0108700 con 0x7fcad00ffba0 2024-02-19T19:05:02.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.730+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7fcac000cba0 con 0x7fcad00ffba0 2024-02-19T19:05:02.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.730+0000 7fcaccff9700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fcabc06ce40 0x7fcabc06f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:02.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.731+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fcac0014070 con 0x7fcad00ffba0 2024-02-19T19:05:02.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.731+0000 7fcac67fc700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fcabc06ce40 0x7fcabc06f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:02.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.731+0000 7fcac67fc700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fcabc06ce40 0x7fcabc06f300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009fd0 tx=0x7fcab8005940 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:02.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.735+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fcac005a1d0 con 0x7fcad00ffba0 2024-02-19T19:05:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.960+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 --> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcad0069520 con 0x7fcabc06ce40 2024-02-19T19:05:02.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.962+0000 7fcaccff9700 1 -- 172.21.15.39:0/3974320638 <== mgr.24415 v2:172.21.15.86:6828/1232386891 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fcad0069520 con 0x7fcabc06ce40 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fcabc06ce40 msgr2=0x7fcabc06f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fcabc06ce40 0x7fcabc06f300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009fd0 tx=0x7fcab8005940 comp rx=0 tx=0).stop 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 msgr2=0x7fcad0196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 0x7fcad0196480 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcac000e8f0 tx=0x7fcac000ecb0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 shutdown_connections 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fcabc06ce40 0x7fcabc06f300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad00ffba0 0x7fcad0196480 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 --2- 172.21.15.39:0/3974320638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcad0100da0 0x7fcad01969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 >> 172.21.15.39:0/3974320638 conn(0x7fcad00fb120 msgr2=0x7fcad0068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 shutdown_connections 2024-02-19T19:05:02.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:02.964+0000 7fcad5645700 1 -- 172.21.15.39:0/3974320638 wait complete. 2024-02-19T19:05:02.995 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 -- 172.21.15.39:0/4078113893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 msgr2=0x7f3034103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 --2- 172.21.15.39:0/4078113893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f3034103b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3024009910 tx=0x7f3024009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 -- 172.21.15.39:0/4078113893 shutdown_connections 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 --2- 172.21.15.39:0/4078113893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 0x7f3034106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 --2- 172.21.15.39:0/4078113893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f3034103b40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 -- 172.21.15.39:0/4078113893 >> 172.21.15.39:0/4078113893 conn(0x7f30340fb160 msgr2=0x7f30340fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 -- 172.21.15.39:0/4078113893 shutdown_connections 2024-02-19T19:05:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.070+0000 7f303a0c2700 1 -- 172.21.15.39:0/4078113893 wait complete. 2024-02-19T19:05:03.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.071+0000 7f303a0c2700 1 Processor -- start 2024-02-19T19:05:03.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.071+0000 7f303a0c2700 1 -- start start 2024-02-19T19:05:03.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.071+0000 7f303a0c2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f30341967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.071+0000 7f303a0c2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 0x7f3034196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.071+0000 7f303a0c2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3034197340 con 0x7f3034101750 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.071+0000 7f303a0c2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3034197480 con 0x7f3034104080 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f30341967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f30341967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36108/0 (socket says 172.21.15.39:36108) 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 -- 172.21.15.39:0/59044973 learned_addr learned my addr 172.21.15.39:0/59044973 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f3032ffd700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 0x7f3034196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 -- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 msgr2=0x7f3034196d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 0x7f3034196d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 -- 172.21.15.39:0/59044973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240095f0 con 0x7f3034101750 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f3032ffd700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 0x7f3034196d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f30337fe700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f30341967e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f302400bd70 tx=0x7f302400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302401d070 con 0x7f3034101750 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303419bed0 con 0x7f3034101750 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3024022470 con 0x7f3034101750 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302400f670 con 0x7f3034101750 2024-02-19T19:05:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.072+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303419c340 con 0x7f3034101750 2024-02-19T19:05:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.073+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3034190630 con 0x7f3034101750 2024-02-19T19:05:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.074+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7f3024022a50 con 0x7f3034101750 2024-02-19T19:05:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.074+0000 7f3030ff9700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f302006ce40 0x7f302006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.075+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f302408de00 con 0x7f3034101750 2024-02-19T19:05:03.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.075+0000 7f3032ffd700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f302006ce40 0x7f302006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.075+0000 7f3032ffd700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f302006ce40 0x7f302006f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f301c005fd0 tx=0x7f301c005dc0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:03.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.079+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f302405b790 con 0x7f3034101750 2024-02-19T19:05:03.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.304+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 --> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30340611d0 con 0x7f302006ce40 2024-02-19T19:05:03.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.305+0000 7f3030ff9700 1 -- 172.21.15.39:0/59044973 <== mgr.24415 v2:172.21.15.86:6828/1232386891 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f30340611d0 con 0x7f302006ce40 2024-02-19T19:05:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f302006ce40 msgr2=0x7f302006f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f302006ce40 0x7f302006f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f301c005fd0 tx=0x7f301c005dc0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 msgr2=0x7f30341967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f30341967e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f302400bd70 tx=0x7f302400be50 comp rx=0 tx=0).stop 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 shutdown_connections 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f302006ce40 0x7f302006f300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3034101750 0x7f30341967e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 --2- 172.21.15.39:0/59044973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3034104080 0x7f3034196d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 >> 172.21.15.39:0/59044973 conn(0x7f30340fb160 msgr2=0x7f30340fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 shutdown_connections 2024-02-19T19:05:03.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.308+0000 7f303a0c2700 1 -- 172.21.15.39:0/59044973 wait complete. 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 -- 172.21.15.39:0/3105657370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8101eb0 msgr2=0x7f06f8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/3105657370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8101eb0 0x7f06f8102310 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009910 tx=0x7f06e8009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 -- 172.21.15.39:0/3105657370 shutdown_connections 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/3105657370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8101eb0 0x7f06f8102310 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/3105657370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f8100cb0 0x7f06f81010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 -- 172.21.15.39:0/3105657370 >> 172.21.15.39:0/3105657370 conn(0x7f06f80fc230 msgr2=0x7f06f80fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 -- 172.21.15.39:0/3105657370 shutdown_connections 2024-02-19T19:05:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.413+0000 7f06fcf4e700 1 -- 172.21.15.39:0/3105657370 wait complete. 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06fcf4e700 1 Processor -- start 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06fcf4e700 1 -- start start 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06fcf4e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 0x7f06f8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06fcf4e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f8101eb0 0x7f06f8198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06fcf4e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f81992d0 con 0x7f06f8100cb0 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06fcf4e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8199410 con 0x7f06f8101eb0 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 0x7f06f8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 0x7f06f8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36138/0 (socket says 172.21.15.39:36138) 2024-02-19T19:05:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.414+0000 7f06f659c700 1 -- 172.21.15.39:0/2523333826 learned_addr learned my addr 172.21.15.39:0/2523333826 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06f5d9b700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f8101eb0 0x7f06f8198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06f659c700 1 -- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f8101eb0 msgr2=0x7f06f8198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06f659c700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f8101eb0 0x7f06f8198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06f659c700 1 -- 172.21.15.39:0/2523333826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e80095f0 con 0x7f06f8100cb0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06f659c700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 0x7f06f8198770 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f06e000b480 tx=0x7f06e000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e0011800 con 0x7f06f8100cb0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06e0011e10 con 0x7f06f8100cb0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f819dec0 con 0x7f06f8100cb0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e0010ea0 con 0x7f06f8100cb0 2024-02-19T19:05:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.415+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f819e380 con 0x7f06f8100cb0 2024-02-19T19:05:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.417+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8073ac0 con 0x7f06f8100cb0 2024-02-19T19:05:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.417+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7f06e0010380 con 0x7f06f8100cb0 2024-02-19T19:05:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.417+0000 7f06ef7fe700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f06e406ce30 0x7f06e406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.417+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f06e008c0d0 con 0x7f06f8100cb0 2024-02-19T19:05:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.418+0000 7f06f5d9b700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f06e406ce30 0x7f06e406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.418+0000 7f06f5d9b700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f06e406ce30 0x7f06e406f2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f06e8006010 tx=0x7f06e800b560 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:03.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.422+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f06e0059920 con 0x7f06f8100cb0 2024-02-19T19:05:03.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.639+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 --> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f06f81060f0 con 0x7f06e406ce30 2024-02-19T19:05:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.651+0000 7f06ef7fe700 1 -- 172.21.15.39:0/2523333826 <== mgr.24415 v2:172.21.15.86:6828/1232386891 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f06f81060f0 con 0x7f06e406ce30 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (7m) 9s ago 8m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (9m) 9s ago 9m 7654k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (7m) 10s ago 7m 7578k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (9m) 9s ago 9m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (7m) 10s ago 7m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (7m) 9s ago 8m 87.5M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (2m) 9s ago 2m 21.4M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (3m) 9s ago 3m 13.2M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (2m) 10s ago 2m 28.7M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (2m) 10s ago 2m 15.0M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:9283,8765,8443 running (10m) 9s ago 10m 361M - 18.2.1 94eadd990e0a f3c738bbd2c8 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (27s) 10s ago 7m 491M - 19.0.0-1450-gf78a58c0 f081f847070a 1d44c757a466 2024-02-19T19:05:03.672 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (10m) 9s ago 10m 50.2M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (7m) 10s ago 7m 39.8M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (8m) 9s ago 8m 14.0M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (7m) 10s ago 7m 15.8M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (6m) 9s ago 6m 51.9M 1216M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (6m) 9s ago 6m 49.5M 1216M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (5m) 9s ago 5m 48.5M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (5m) 10s ago 5m 47.5M 4970M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (4m) 10s ago 4m 45.9M 4970M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (4m) 10s ago 4m 47.8M 4970M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:05:03.673 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (6m) 9s ago 8m 62.0M - 2.43.0 a07b618ecd1d 54ef95da7db1 2024-02-19T19:05:03.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f06e406ce30 msgr2=0x7f06e406f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f06e406ce30 0x7f06e406f2f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f06e8006010 tx=0x7f06e800b560 comp rx=0 tx=0).stop 2024-02-19T19:05:03.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 msgr2=0x7f06f8198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 0x7f06f8198770 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f06e000b480 tx=0x7f06e000b790 comp rx=0 tx=0).stop 2024-02-19T19:05:03.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 shutdown_connections 2024-02-19T19:05:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f06e406ce30 0x7f06e406f2f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f8100cb0 0x7f06f8198770 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 --2- 172.21.15.39:0/2523333826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f8101eb0 0x7f06f8198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.653+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 >> 172.21.15.39:0/2523333826 conn(0x7f06f80fc230 msgr2=0x7f06f80fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.654+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 shutdown_connections 2024-02-19T19:05:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.654+0000 7f06fcf4e700 1 -- 172.21.15.39:0/2523333826 wait complete. 2024-02-19T19:05:03.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:03 smithi039 ceph-mon[28988]: pgmap v12: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 -- 172.21.15.39:0/1950800890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3280fef60 msgr2=0x7fe3280ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 --2- 172.21.15.39:0/1950800890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3280fef60 0x7fe3280ff380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe324009910 tx=0x7fe324009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 -- 172.21.15.39:0/1950800890 shutdown_connections 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 --2- 172.21.15.39:0/1950800890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe328101f40 0x7fe3281023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 --2- 172.21.15.39:0/1950800890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3280fef60 0x7fe3280ff380 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 -- 172.21.15.39:0/1950800890 >> 172.21.15.39:0/1950800890 conn(0x7fe3280fa8c0 msgr2=0x7fe3280fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 -- 172.21.15.39:0/1950800890 shutdown_connections 2024-02-19T19:05:03.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.758+0000 7fe32f565700 1 -- 172.21.15.39:0/1950800890 wait complete. 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32f565700 1 Processor -- start 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32f565700 1 -- start start 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32f565700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3280fef60 0x7fe3281018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32f565700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 0x7fe3280ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32f565700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328100430 con 0x7fe328101f40 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32f565700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328100570 con 0x7fe3280fef60 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32cb00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 0x7fe3280ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32cb00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 0x7fe3280ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36160/0 (socket says 172.21.15.39:36160) 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32cb00700 1 -- 172.21.15.39:0/3158420932 learned_addr learned my addr 172.21.15.39:0/3158420932 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.759+0000 7fe32d301700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3280fef60 0x7fe3281018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe32cb00700 1 -- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3280fef60 msgr2=0x7fe3281018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe32cb00700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3280fef60 0x7fe3281018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe32cb00700 1 -- 172.21.15.39:0/3158420932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3240095f0 con 0x7fe328101f40 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe32cb00700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 0x7fe3280ffef0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe31800c930 tx=0x7fe31800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe318007ad0 con 0x7fe328101f40 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe31800ce80 con 0x7fe328101f40 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe328100850 con 0x7fe328101f40 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe318018630 con 0x7fe328101f40 2024-02-19T19:05:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.760+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32819e510 con 0x7fe328101f40 2024-02-19T19:05:03.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.761+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe328109800 con 0x7fe328101f40 2024-02-19T19:05:03.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.762+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7fe318007c30 con 0x7fe328101f40 2024-02-19T19:05:03.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.762+0000 7fe31e7fc700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fe31406ce40 0x7fe31406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:03.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.762+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fe31808c530 con 0x7fe328101f40 2024-02-19T19:05:03.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.762+0000 7fe32d301700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fe31406ce40 0x7fe31406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:03.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.763+0000 7fe32d301700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fe31406ce40 0x7fe31406f300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe324006010 tx=0x7fe324005280 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:03.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:03 smithi086 ceph-mon[37629]: pgmap v12: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:05:03.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:03.767+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fe318059cf0 con 0x7fe328101f40 2024-02-19T19:05:04.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.066+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe3281099c0 con 0x7fe328101f40 2024-02-19T19:05:04.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.067+0000 7fe31e7fc700 1 -- 172.21.15.39:0/3158420932 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7fe318059880 con 0x7fe328101f40 2024-02-19T19:05:04.087 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:05:04.087 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:04.088 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:05:04.089 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-02-19T19:05:04.089 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:05:04.089 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:05:04.089 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:05:04.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fe31406ce40 msgr2=0x7fe31406f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fe31406ce40 0x7fe31406f300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe324006010 tx=0x7fe324005280 comp rx=0 tx=0).stop 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 msgr2=0x7fe3280ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 0x7fe3280ffef0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe31800c930 tx=0x7fe31800cc40 comp rx=0 tx=0).stop 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 shutdown_connections 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fe31406ce40 0x7fe31406f300 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3280fef60 0x7fe3281018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 --2- 172.21.15.39:0/3158420932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe328101f40 0x7fe3280ffef0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 >> 172.21.15.39:0/3158420932 conn(0x7fe3280fa8c0 msgr2=0x7fe328105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 shutdown_connections 2024-02-19T19:05:04.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.069+0000 7fe32f565700 1 -- 172.21.15.39:0/3158420932 wait complete. 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3716091742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 msgr2=0x7f2e40106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3716091742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40106520 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009910 tx=0x7f2e2c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3716091742 shutdown_connections 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3716091742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40106520 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3716091742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e40101800 0x7f2e40103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3716091742 >> 172.21.15.39:0/3716091742 conn(0x7f2e400fb190 msgr2=0x7f2e400fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3716091742 shutdown_connections 2024-02-19T19:05:04.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.177+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3716091742 wait complete. 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e46f7f700 1 Processor -- start 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e46f7f700 1 -- start start 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e46f7f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e40101800 0x7f2e40196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e46f7f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e46f7f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e40196fb0 con 0x7f2e40104130 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e46f7f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e401970f0 con 0x7f2e40101800 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e3ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e3ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36180/0 (socket says 172.21.15.39:36180) 2024-02-19T19:05:04.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e44d1b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e40101800 0x7f2e40196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.178+0000 7f2e3ffff700 1 -- 172.21.15.39:0/3669091239 learned_addr learned my addr 172.21.15.39:0/3669091239 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3ffff700 1 -- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e40101800 msgr2=0x7f2e40196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3ffff700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e40101800 0x7f2e40196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3ffff700 1 -- 172.21.15.39:0/3669091239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0095f0 con 0x7f2e40104130 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3ffff700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40196990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c000c00 tx=0x7f2e2c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e2c01d070 con 0x7f2e40104130 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4019bb40 con 0x7f2e40104130 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2e2c005dc0 con 0x7f2e40104130 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e2c0226a0 con 0x7f2e40104130 2024-02-19T19:05:04.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.179+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4019c030 con 0x7f2e40104130 2024-02-19T19:05:04.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.180+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e40190610 con 0x7f2e40104130 2024-02-19T19:05:04.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.181+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7f2e2c0038f0 con 0x7f2e40104130 2024-02-19T19:05:04.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.181+0000 7f2e3dffb700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f2e3006ce30 0x7f2e3006f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.181+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f2e2c08db40 con 0x7f2e40104130 2024-02-19T19:05:04.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.181+0000 7f2e44d1b700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f2e3006ce30 0x7f2e3006f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.182+0000 7f2e44d1b700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f2e3006ce30 0x7f2e3006f2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009c60 tx=0x7f2e34009260 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:04.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.186+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f2e2c05b280 con 0x7f2e40104130 2024-02-19T19:05:04.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.443+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2e4002cc70 con 0x7f2e40104130 2024-02-19T19:05:04.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.444+0000 7f2e3dffb700 1 -- 172.21.15.39:0/3669091239 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1847 (secure 0 0 0) 0x7f2e2c003680 con 0x7f2e40104130 2024-02-19T19:05:04.464 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:05:04.465 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:05:04.466 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:04.467 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:04.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f2e3006ce30 msgr2=0x7f2e3006f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f2e3006ce30 0x7f2e3006f2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009c60 tx=0x7f2e34009260 comp rx=0 tx=0).stop 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 msgr2=0x7f2e40196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40196990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c000c00 tx=0x7f2e2c00f690 comp rx=0 tx=0).stop 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 shutdown_connections 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7f2e3006ce30 0x7f2e3006f2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e40101800 0x7f2e40196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 --2- 172.21.15.39:0/3669091239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e40104130 0x7f2e40196990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 >> 172.21.15.39:0/3669091239 conn(0x7f2e400fb190 msgr2=0x7f2e400fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 shutdown_connections 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.446+0000 7f2e46f7f700 1 -- 172.21.15.39:0/3669091239 wait complete. 2024-02-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:05:04.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 -- 172.21.15.39:0/2764043470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e00ffbc0 msgr2=0x7ff1e00fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/2764043470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e00ffbc0 0x7ff1e00fffe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009910 tx=0x7ff1d0009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:04.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 -- 172.21.15.39:0/2764043470 shutdown_connections 2024-02-19T19:05:04.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/2764043470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e0100dc0 0x7ff1e0101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/2764043470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e00ffbc0 0x7ff1e00fffe0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 -- 172.21.15.39:0/2764043470 >> 172.21.15.39:0/2764043470 conn(0x7ff1e00fb160 msgr2=0x7ff1e00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 -- 172.21.15.39:0/2764043470 shutdown_connections 2024-02-19T19:05:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.557+0000 7ff1e56b9700 1 -- 172.21.15.39:0/2764043470 wait complete. 2024-02-19T19:05:04.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.558+0000 7ff1e56b9700 1 Processor -- start 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.558+0000 7ff1e56b9700 1 -- start start 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.558+0000 7ff1e56b9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e00ffbc0 0x7ff1e00699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.558+0000 7ff1e56b9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 0x7ff1e019b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.558+0000 7ff1e56b9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e019bb00 con 0x7ff1e0100dc0 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.558+0000 7ff1e56b9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e019bc40 con 0x7ff1e00ffbc0 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 0x7ff1e019b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1deffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e00ffbc0 0x7ff1e00699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 0x7ff1e019b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36196/0 (socket says 172.21.15.39:36196) 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 -- 172.21.15.39:0/582770504 learned_addr learned my addr 172.21.15.39:0/582770504 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 -- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e00ffbc0 msgr2=0x7ff1e00699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e00ffbc0 0x7ff1e00699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 -- 172.21.15.39:0/582770504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d00095f0 con 0x7ff1e0100dc0 2024-02-19T19:05:04.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1de7fc700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 0x7ff1e019b4e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400d900 tx=0x7ff1d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:04.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1d40041f0 con 0x7ff1e0100dc0 2024-02-19T19:05:04.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff1d4004ce0 con 0x7ff1e0100dc0 2024-02-19T19:05:04.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e019bf20 con 0x7ff1e0100dc0 2024-02-19T19:05:04.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1d400b750 con 0x7ff1e0100dc0 2024-02-19T19:05:04.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.559+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e019c440 con 0x7ff1e0100dc0 2024-02-19T19:05:04.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.561+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e018f530 con 0x7ff1e0100dc0 2024-02-19T19:05:04.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.561+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7ff1d4004e50 con 0x7ff1e0100dc0 2024-02-19T19:05:04.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.561+0000 7ff1c7fff700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7ff1c806ce40 0x7ff1c806f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.561+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7ff1d408c460 con 0x7ff1e0100dc0 2024-02-19T19:05:04.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.561+0000 7ff1deffd700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7ff1c806ce40 0x7ff1c806f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.562+0000 7ff1deffd700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7ff1c806ce40 0x7ff1c806f300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0006010 tx=0x7ff1d0005ec0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:04.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.566+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7ff1d4059c20 con 0x7ff1e0100dc0 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: Adjusting osd_memory_target on smithi086 to 2240M 2024-02-19T19:05:04.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3158420932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: pgmap v13: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:04.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:04 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:04.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:04.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:04.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:04.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: Adjusting osd_memory_target on smithi086 to 2240M 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3158420932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:04.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:04.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: pgmap v13: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:04.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:04 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:04.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.789+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 --> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1e00611d0 con 0x7ff1c806ce40 2024-02-19T19:05:04.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.791+0000 7ff1c7fff700 1 -- 172.21.15.39:0/582770504 <== mgr.24415 v2:172.21.15.86:6828/1232386891 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7ff1e00611d0 con 0x7ff1c806ce40 2024-02-19T19:05:04.811 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:05:04.811 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:05:04.811 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:05:04.812 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:05:04.812 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [], 2024-02-19T19:05:04.812 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "1/23 daemons upgraded", 2024-02-19T19:05:04.812 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "", 2024-02-19T19:05:04.812 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:05:04.812 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7ff1c806ce40 msgr2=0x7ff1c806f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7ff1c806ce40 0x7ff1c806f300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0006010 tx=0x7ff1d0005ec0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 msgr2=0x7ff1e019b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 0x7ff1e019b4e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400d900 tx=0x7ff1d400dc10 comp rx=0 tx=0).stop 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 shutdown_connections 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7ff1c806ce40 0x7ff1c806f300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1e00ffbc0 0x7ff1e00699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 --2- 172.21.15.39:0/582770504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e0100dc0 0x7ff1e019b4e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 >> 172.21.15.39:0/582770504 conn(0x7ff1e00fb160 msgr2=0x7ff1e01040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 shutdown_connections 2024-02-19T19:05:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.793+0000 7ff1e56b9700 1 -- 172.21.15.39:0/582770504 wait complete. 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/4288696121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 msgr2=0x7fa8d0106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/4288696121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d0106290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4009910 tx=0x7fa8c4009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/4288696121 shutdown_connections 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/4288696121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d0106290 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/4288696121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8d0101570 0x7fa8d0103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/4288696121 >> 172.21.15.39:0/4288696121 conn(0x7fa8d00faf80 msgr2=0x7fa8d00fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/4288696121 shutdown_connections 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/4288696121 wait complete. 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 Processor -- start 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 -- start start 2024-02-19T19:05:04.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.900+0000 7fa8d5fa0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8d0101570 0x7fa8d0196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8d5fa0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d01967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8d5fa0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0196de0 con 0x7fa8d0103ea0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8d5fa0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0196f20 con 0x7fa8d0101570 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d01967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d01967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36226/0 (socket says 172.21.15.39:36226) 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 -- 172.21.15.39:0/2345587653 learned_addr learned my addr 172.21.15.39:0/2345587653 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8d4f9e700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8d0101570 0x7fa8d0196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 -- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8d0101570 msgr2=0x7fa8d0196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8d0101570 0x7fa8d0196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 -- 172.21.15.39:0/2345587653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40095f0 con 0x7fa8d0103ea0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cffff700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d01967c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400bd70 tx=0x7fa8c400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8c401d070 con 0x7fa8d0103ea0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa8c4005dc0 con 0x7fa8d0103ea0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d019b970 con 0x7fa8d0103ea0 2024-02-19T19:05:04.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.901+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8c4021410 con 0x7fa8d0103ea0 2024-02-19T19:05:04.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.902+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d019be60 con 0x7fa8d0103ea0 2024-02-19T19:05:04.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.902+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d0190450 con 0x7fa8d0103ea0 2024-02-19T19:05:04.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.903+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 25) v1 ==== 90931+0+0 (secure 0 0 0) 0x7fa8c400f7d0 con 0x7fa8d0103ea0 2024-02-19T19:05:04.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.903+0000 7fa8cdffb700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fa8b806ce40 0x7fa8b806f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:04.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.904+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fa8c408d860 con 0x7fa8d0103ea0 2024-02-19T19:05:04.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.904+0000 7fa8d4f9e700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fa8b806ce40 0x7fa8b806f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:04.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.904+0000 7fa8d4f9e700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fa8b806ce40 0x7fa8b806f300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0007c60 tx=0x7fa8c00073d0 comp rx=0 tx=0).ready entity=mgr.24415 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:04.908+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fa8c405afa0 con 0x7fa8d0103ea0 2024-02-19T19:05:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.206+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa8d002ce80 con 0x7fa8d0103ea0 2024-02-19T19:05:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.207+0000 7fa8cdffb700 1 -- 172.21.15.39:0/2345587653 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fa8c400f460 con 0x7fa8d0103ea0 2024-02-19T19:05:05.228 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:05.228 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:05.228 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fa8b806ce40 msgr2=0x7fa8b806f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fa8b806ce40 0x7fa8b806f300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0007c60 tx=0x7fa8c00073d0 comp rx=0 tx=0).stop 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 msgr2=0x7fa8d01967c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d01967c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400bd70 tx=0x7fa8c400bda0 comp rx=0 tx=0).stop 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 shutdown_connections 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:6828/1232386891,v1:172.21.15.86:6829/1232386891] conn(0x7fa8b806ce40 0x7fa8b806f300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8d0101570 0x7fa8d0196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 --2- 172.21.15.39:0/2345587653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa8d0103ea0 0x7fa8d01967c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 >> 172.21.15.39:0/2345587653 conn(0x7fa8d00faf80 msgr2=0x7fa8d00fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 shutdown_connections 2024-02-19T19:05:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:05.209+0000 7fa8d5fa0700 1 -- 172.21.15.39:0/2345587653 wait complete. 2024-02-19T19:05:05.496 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/3669091239' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: Reconfiguring prometheus.smithi039 (dependencies changed)... 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2345587653' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:05.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:05 smithi039 ceph-mon[28988]: Reconfiguring daemon prometheus.smithi039 on smithi039 2024-02-19T19:05:05.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3669091239' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: Reconfiguring prometheus.smithi039 (dependencies changed)... 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2345587653' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:05.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:05 smithi086 ceph-mon[37629]: Reconfiguring daemon prometheus.smithi039 on smithi039 2024-02-19T19:05:06.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:06 smithi039 ceph-mon[28988]: pgmap v14: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:06.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:06 smithi086 ceph-mon[37629]: pgmap v14: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:09.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:09 smithi086 ceph-mon[37629]: pgmap v15: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:09.473 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:09 smithi039 ceph-mon[28988]: pgmap v15: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:11.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:11 smithi086 ceph-mon[37629]: pgmap v16: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:11.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:11 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:11.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:11 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:11.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:11 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:11.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:11 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:11.573 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:11 smithi039 ceph-mon[28988]: pgmap v16: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:11.573 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:11 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:11.573 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:11 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:11.573 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:11 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:11.573 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:11 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:12.368 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:12 smithi039 ceph-mon[28988]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:12.368 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:12 smithi039 ceph-mon[28988]: Upgrade: Need to upgrade myself (mgr.smithi086.huubbu) 2024-02-19T19:05:12.369 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:12 smithi039 ceph-mon[28988]: Upgrade: Need to upgrade myself (mgr.smithi086.huubbu) 2024-02-19T19:05:12.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:12 smithi086 ceph-mon[37629]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:12.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:12 smithi086 ceph-mon[37629]: Upgrade: Need to upgrade myself (mgr.smithi086.huubbu) 2024-02-19T19:05:12.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:12 smithi086 ceph-mon[37629]: Upgrade: Need to upgrade myself (mgr.smithi086.huubbu) 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: pgmap v17: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: Upgrade: Updating mgr.smithi039.ydmpuh 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:13.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:13 smithi039 ceph-mon[28988]: Deploying daemon mgr.smithi039.ydmpuh on smithi039 2024-02-19T19:05:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: pgmap v17: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: Upgrade: Updating mgr.smithi039.ydmpuh 2024-02-19T19:05:13.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:05:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:05:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:05:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:13 smithi086 ceph-mon[37629]: Deploying daemon mgr.smithi039.ydmpuh on smithi039 2024-02-19T19:05:15.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:15 smithi039 ceph-mon[28988]: pgmap v18: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:15.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:15 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:15.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:15 smithi086 ceph-mon[37629]: pgmap v18: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:15.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:15 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:17.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:17 smithi039 ceph-mon[28988]: pgmap v19: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:17.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:17 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:17.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:17 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:17.582 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:17 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:17.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:17 smithi086 ceph-mon[37629]: pgmap v19: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:17.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:17 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:17.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:17 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:17.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:17 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:19.746 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:19 smithi039 ceph-mon[28988]: pgmap v20: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:19.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:19 smithi086 ceph-mon[37629]: pgmap v20: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:21.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:21 smithi039 ceph-mon[28988]: pgmap v21: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:21.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:21 smithi086 ceph-mon[37629]: pgmap v21: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:23.378 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:23 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:23.378 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:23 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:23.378 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:23 smithi039 ceph-mon[28988]: pgmap v22: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:23.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:23 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:23.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:23 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:23.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:23 smithi086 ceph-mon[37629]: pgmap v22: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: Standby manager daemon smithi039.ydmpuh restarted 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: Standby manager daemon smithi039.ydmpuh started 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/crt"}]: dispatch 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:05:24.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/key"}]: dispatch 2024-02-19T19:05:24.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:24 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:05:24.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: Standby manager daemon smithi039.ydmpuh restarted 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: Standby manager daemon smithi039.ydmpuh started 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/crt"}]: dispatch 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi039.ydmpuh/key"}]: dispatch 2024-02-19T19:05:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:24 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:05:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:25 smithi039 ceph-mon[28988]: pgmap v23: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:25 smithi039 ceph-mon[28988]: mgrmap e26: smithi086.huubbu(active, since 40s), standbys: smithi039.ydmpuh 2024-02-19T19:05:25.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:25 smithi086 ceph-mon[37629]: pgmap v23: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:25.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:25 smithi086 ceph-mon[37629]: mgrmap e26: smithi086.huubbu(active, since 40s), standbys: smithi039.ydmpuh 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: pgmap v24: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr fail", "who": "smithi086.huubbu"}]: dispatch 2024-02-19T19:05:27.309 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr fail", "who": "smithi086.huubbu"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd='[{"prefix": "mgr fail", "who": "smithi086.huubbu"}]': finished 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: mgrmap e27: smithi039.ydmpuh(active, starting, since 0.00750141s) 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:05:27.310 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:05:27.311 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:27 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T19:05:27.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: pgmap v24: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:27.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:27.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 172.21.15.86:0/132239416' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr fail", "who": "smithi086.huubbu"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "mgr fail", "who": "smithi086.huubbu"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.24415 ' entity='mgr.smithi086.huubbu' cmd='[{"prefix": "mgr fail", "who": "smithi086.huubbu"}]': finished 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: mgrmap e27: smithi039.ydmpuh(active, starting, since 0.00750141s) 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:05:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:27.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:05:27.500 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:27 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: [19/Feb/2024:19:05:27] ENGINE Bus STARTING 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: [19/Feb/2024:19:05:27] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: [19/Feb/2024:19:05:27] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: [19/Feb/2024:19:05:27] ENGINE Bus STARTED 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: mgrmap e28: smithi039.ydmpuh(active, since 1.01204s) 2024-02-19T19:05:29.034 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:28 smithi039 ceph-mon[28988]: pgmap v3: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:29.210 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: [19/Feb/2024:19:05:27] ENGINE Bus STARTING 2024-02-19T19:05:29.210 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: [19/Feb/2024:19:05:27] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T19:05:29.210 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: [19/Feb/2024:19:05:27] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T19:05:29.210 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: [19/Feb/2024:19:05:27] ENGINE Bus STARTED 2024-02-19T19:05:29.210 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:29.211 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:29.211 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: mgrmap e28: smithi039.ydmpuh(active, since 1.01204s) 2024-02-19T19:05:29.211 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:28 smithi086 ceph-mon[37629]: pgmap v3: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:30.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:30 smithi086 ceph-mon[37629]: pgmap v4: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:30.416 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:30 smithi039 ceph-mon[28988]: pgmap v4: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:31.328 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:31 smithi039 ceph-mon[28988]: mgrmap e29: smithi039.ydmpuh(active, since 3s) 2024-02-19T19:05:31.328 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:31 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:31.328 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:31 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:31.472 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:31 smithi086 ceph-mon[37629]: mgrmap e29: smithi039.ydmpuh(active, since 3s) 2024-02-19T19:05:31.472 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:31 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:31.473 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:31 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:32.222 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:32 smithi086 ceph-mon[37629]: pgmap v5: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:32.222 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:32 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:32.222 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:32 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:32.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:32 smithi039 ceph-mon[28988]: pgmap v5: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:32.246 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:32 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:32.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:32 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:34.187 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:34 smithi086 ceph-mon[37629]: pgmap v6: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:34.187 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:34 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:34.187 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:34 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:34.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:34 smithi039 ceph-mon[28988]: pgmap v6: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:05:34.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:34 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:34.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:34 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:35.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.329+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/1066688489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb01011c0 msgr2=0x7f6cb01015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.329+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/1066688489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb01011c0 0x7f6cb01015e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009910 tx=0x7f6cac009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:35.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.329+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/1066688489 shutdown_connections 2024-02-19T19:05:35.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.329+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/1066688489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb00ffe20 0x7f6cb01002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.329+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/1066688489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb01011c0 0x7f6cb01015e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.329+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/1066688489 >> 172.21.15.39:0/1066688489 conn(0x7f6cb00fa8b0 msgr2=0x7f6cb00fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:35.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.330+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/1066688489 shutdown_connections 2024-02-19T19:05:35.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.330+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/1066688489 wait complete. 2024-02-19T19:05:35.352 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:05:35.352 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:05:35.352 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:05:35.352 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:05:35.353 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:05:35.353 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:35.353 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:35 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:35.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.330+0000 7f6cb7ab8700 1 Processor -- start 2024-02-19T19:05:35.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.330+0000 7f6cb7ab8700 1 -- start start 2024-02-19T19:05:35.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.330+0000 7f6cb7ab8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 0x7f6cb00ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:35.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb7ab8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb01011c0 0x7f6cb019b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:35.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb7ab8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb019b840 con 0x7f6cb00ffe20 2024-02-19T19:05:35.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb7ab8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb019b980 con 0x7f6cb01011c0 2024-02-19T19:05:35.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 0x7f6cb00ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:35.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 0x7f6cb00ff8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58430/0 (socket says 172.21.15.39:58430) 2024-02-19T19:05:35.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 -- 172.21.15.39:0/2394377579 learned_addr learned my addr 172.21.15.39:0/2394377579 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:35.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5053700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb01011c0 0x7f6cb019b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:35.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 -- 172.21.15.39:0/2394377579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb01011c0 msgr2=0x7f6cb019b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb01011c0 0x7f6cb019b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 -- 172.21.15.39:0/2394377579 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac0095f0 con 0x7f6cb00ffe20 2024-02-19T19:05:35.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb5854700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 0x7f6cb00ff8d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6cac000c00 tx=0x7f6cac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:35.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cac01d070 con 0x7f6cb00ffe20 2024-02-19T19:05:35.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb019bc00 con 0x7f6cb00ffe20 2024-02-19T19:05:35.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6cac005dc0 con 0x7f6cb00ffe20 2024-02-19T19:05:35.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cac022470 con 0x7f6cb00ffe20 2024-02-19T19:05:35.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.331+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb019c0f0 con 0x7f6cb00ffe20 2024-02-19T19:05:35.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.333+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb0108740 con 0x7f6cb00ffe20 2024-02-19T19:05:35.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.333+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7f6cac003810 con 0x7f6cb00ffe20 2024-02-19T19:05:35.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.333+0000 7f6ca6ffd700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f6c9c06d660 0x7f6c9c06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:35.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.333+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6cac08e6e0 con 0x7f6cb00ffe20 2024-02-19T19:05:35.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.333+0000 7f6cb5053700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f6c9c06d660 0x7f6c9c06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:35.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.334+0000 7f6cb5053700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f6c9c06d660 0x7f6c9c06fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009c60 tx=0x7f6ca0009260 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:35.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.338+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f6cac05bcd0 con 0x7f6cb00ffe20 2024-02-19T19:05:35.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:05:35.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:05:35.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:05:35.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:05:35.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/855808531' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:05:35.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:35.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:35 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:35.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.562+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 --> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cb0105780 con 0x7f6c9c06d660 2024-02-19T19:05:35.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.563+0000 7f6ca6ffd700 1 -- 172.21.15.39:0/2394377579 <== mgr.14744 v2:172.21.15.39:6800/913229036 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f6cb0105780 con 0x7f6c9c06d660 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f6c9c06d660 msgr2=0x7f6c9c06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f6c9c06d660 0x7f6c9c06fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009c60 tx=0x7f6ca0009260 comp rx=0 tx=0).stop 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 msgr2=0x7f6cb00ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 0x7f6cb00ff8d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6cac000c00 tx=0x7f6cac00f690 comp rx=0 tx=0).stop 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 shutdown_connections 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f6c9c06d660 0x7f6c9c06fb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb00ffe20 0x7f6cb00ff8d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 --2- 172.21.15.39:0/2394377579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb01011c0 0x7f6cb019b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 >> 172.21.15.39:0/2394377579 conn(0x7f6cb00fa8b0 msgr2=0x7f6cb0104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:35.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 shutdown_connections 2024-02-19T19:05:35.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.565+0000 7f6cb7ab8700 1 -- 172.21.15.39:0/2394377579 wait complete. 2024-02-19T19:05:35.597 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:05:35.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 -- 172.21.15.39:0/909609552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 msgr2=0x7f2ee80ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 --2- 172.21.15.39:0/909609552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee80ff160 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009910 tx=0x7f2edc009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:35.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 -- 172.21.15.39:0/909609552 shutdown_connections 2024-02-19T19:05:35.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 --2- 172.21.15.39:0/909609552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee80ff160 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 --2- 172.21.15.39:0/909609552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 0x7f2ee81026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 -- 172.21.15.39:0/909609552 >> 172.21.15.39:0/909609552 conn(0x7f2ee80fa8c0 msgr2=0x7f2ee80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:35.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.678+0000 7f2eecc49700 1 -- 172.21.15.39:0/909609552 shutdown_connections 2024-02-19T19:05:35.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.679+0000 7f2eecc49700 1 -- 172.21.15.39:0/909609552 wait complete. 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.679+0000 7f2eecc49700 1 Processor -- start 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.679+0000 7f2eecc49700 1 -- start start 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2eecc49700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee8101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2eecc49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 0x7f2ee80fffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2eecc49700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8100510 con 0x7f2ee80fed00 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2eecc49700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8100650 con 0x7f2ee8102280 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee8101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee8101980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58450/0 (socket says 172.21.15.39:58450) 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 -- 172.21.15.39:0/738356139 learned_addr learned my addr 172.21.15.39:0/738356139 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee5d9b700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 0x7f2ee80fffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:35.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 -- 172.21.15.39:0/738356139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 msgr2=0x7f2ee80fffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 0x7f2ee80fffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 -- 172.21.15.39:0/738356139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0095f0 con 0x7f2ee80fed00 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee5d9b700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 0x7f2ee80fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ee659c700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee8101980 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800e8f0 tx=0x7f2ed800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed8009790 con 0x7f2ee80fed00 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2ed800c810 con 0x7f2ee80fed00 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.680+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed8003d20 con 0x7f2ee80fed00 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.681+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee8100930 con 0x7f2ee80fed00 2024-02-19T19:05:35.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.681+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee819e5b0 con 0x7f2ee80fed00 2024-02-19T19:05:35.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.681+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee8109810 con 0x7f2ee80fed00 2024-02-19T19:05:35.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.682+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7f2ed80041f0 con 0x7f2ee80fed00 2024-02-19T19:05:35.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.683+0000 7f2ed77fe700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f2ed006d660 0x7f2ed006fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:35.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.683+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2ed8014070 con 0x7f2ee80fed00 2024-02-19T19:05:35.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.683+0000 7f2ee5d9b700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f2ed006d660 0x7f2ed006fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:35.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.683+0000 7f2ee5d9b700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f2ed006d660 0x7f2ed006fb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009fd0 tx=0x7f2edc005940 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:35.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.691+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f2ed805aa90 con 0x7f2ee80fed00 2024-02-19T19:05:35.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.923+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 --> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ee819eb60 con 0x7f2ed006d660 2024-02-19T19:05:35.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.924+0000 7f2ed77fe700 1 -- 172.21.15.39:0/738356139 <== mgr.14744 v2:172.21.15.39:6800/913229036 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f2ee819eb60 con 0x7f2ed006d660 2024-02-19T19:05:35.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f2ed006d660 msgr2=0x7f2ed006fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f2ed006d660 0x7f2ed006fb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009fd0 tx=0x7f2edc005940 comp rx=0 tx=0).stop 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 msgr2=0x7f2ee8101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee8101980 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800e8f0 tx=0x7f2ed800ec00 comp rx=0 tx=0).stop 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 shutdown_connections 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f2ed006d660 0x7f2ed006fb20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ee80fed00 0x7f2ee8101980 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 --2- 172.21.15.39:0/738356139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ee8102280 0x7f2ee80fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.926+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 >> 172.21.15.39:0/738356139 conn(0x7f2ee80fa8c0 msgr2=0x7f2ee8105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.927+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 shutdown_connections 2024-02-19T19:05:35.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:35.927+0000 7f2eecc49700 1 -- 172.21.15.39:0/738356139 wait complete. 2024-02-19T19:05:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 -- 172.21.15.39:0/1406803733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab80ffbb0 msgr2=0x7faab80fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 --2- 172.21.15.39:0/1406803733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab80ffbb0 0x7faab80fffd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009910 tx=0x7faaa8009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 -- 172.21.15.39:0/1406803733 shutdown_connections 2024-02-19T19:05:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 --2- 172.21.15.39:0/1406803733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab8100db0 0x7faab8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 --2- 172.21.15.39:0/1406803733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab80ffbb0 0x7faab80fffd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 -- 172.21.15.39:0/1406803733 >> 172.21.15.39:0/1406803733 conn(0x7faab80fb150 msgr2=0x7faab80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:36.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.036+0000 7faabe3e7700 1 -- 172.21.15.39:0/1406803733 shutdown_connections 2024-02-19T19:05:36.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 -- 172.21.15.39:0/1406803733 wait complete. 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 Processor -- start 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 -- start start 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab80ffbb0 0x7faab81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 0x7faab8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab8197020 con 0x7faab8100db0 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.037+0000 7faabe3e7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab8197160 con 0x7faab80ffbb0 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 0x7faab8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 0x7faab8196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58462/0 (socket says 172.21.15.39:58462) 2024-02-19T19:05:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 -- 172.21.15.39:0/3542066495 learned_addr learned my addr 172.21.15.39:0/3542066495 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:36.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab7fff700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab80ffbb0 0x7faab81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 -- 172.21.15.39:0/3542066495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab80ffbb0 msgr2=0x7faab81964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab80ffbb0 0x7faab81964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 -- 172.21.15.39:0/3542066495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80095f0 con 0x7faab8100db0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.038+0000 7faab77fe700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 0x7faab8196a00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faaac00d8d0 tx=0x7faaac00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.039+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac0098a0 con 0x7faab8100db0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.039+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab819bc10 con 0x7faab8100db0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.039+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faaac010460 con 0x7faab8100db0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.039+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faaac00f5f0 con 0x7faab8100db0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.039+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab819c160 con 0x7faab8100db0 2024-02-19T19:05:36.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.040+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab8108710 con 0x7faab8100db0 2024-02-19T19:05:36.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.041+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7faaac009a00 con 0x7faab8100db0 2024-02-19T19:05:36.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.041+0000 7faab57fa700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7faaa006d660 0x7faaa006fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.041+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7faaac08d0b0 con 0x7faab8100db0 2024-02-19T19:05:36.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.041+0000 7faab7fff700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7faaa006d660 0x7faaa006fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.042+0000 7faab7fff700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7faaa006d660 0x7faaa006fb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faaa8006010 tx=0x7faaa800b540 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:36.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.050+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7faaac05a6a0 con 0x7faab8100db0 2024-02-19T19:05:36.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:36 smithi039 ceph-mon[28988]: pgmap v7: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-19T19:05:36.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:36 smithi039 ceph-mon[28988]: mgrmap e30: smithi039.ydmpuh(active, since 8s), standbys: smithi086.huubbu 2024-02-19T19:05:36.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:36 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T19:05:36.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.266+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 --> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faab8069520 con 0x7faaa006d660 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.277+0000 7faab57fa700 1 -- 172.21.15.39:0/3542066495 <== mgr.14744 v2:172.21.15.39:6800/913229036 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7faab8069520 con 0x7faaa006d660 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (7m) 2s ago 9m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (9m) 2s ago 9m 7700k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (8m) 5s ago 8m 7625k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (9m) 2s ago 9m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (8m) 5s ago 8m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:05:36.297 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (7m) 2s ago 8m 85.6M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (3m) 2s ago 3m 22.9M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (3m) 2s ago 3m 13.6M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (3m) 5s ago 3m 28.8M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (3m) 5s ago 3m 15.3M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (20s) 2s ago 10m 475M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (60s) 5s ago 8m 70.3M - 19.0.0-1450-gf78a58c0 f081f847070a 1d44c757a466 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (10m) 2s ago 10m 53.3M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (8m) 5s ago 8m 39.4M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (9m) 2s ago 9m 14.0M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (8m) 5s ago 8m 16.4M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (7m) 2s ago 7m 52.4M 1216M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (6m) 2s ago 6m 49.9M 1216M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (6m) 2s ago 6m 48.8M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (5m) 5s ago 5m 48.3M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:05:36.298 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (5m) 5s ago 5m 46.5M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:05:36.299 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (5m) 5s ago 4m 48.3M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:05:36.299 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (26s) 2s ago 8m 42.4M - 2.43.0 a07b618ecd1d ace1f38a9575 2024-02-19T19:05:36.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7faaa006d660 msgr2=0x7faaa006fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7faaa006d660 0x7faaa006fb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faaa8006010 tx=0x7faaa800b540 comp rx=0 tx=0).stop 2024-02-19T19:05:36.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 msgr2=0x7faab8196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 0x7faab8196a00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faaac00d8d0 tx=0x7faaac00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 shutdown_connections 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faab80ffbb0 0x7faab81964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7faaa006d660 0x7faaa006fb20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 --2- 172.21.15.39:0/3542066495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8100db0 0x7faab8196a00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.279+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 >> 172.21.15.39:0/3542066495 conn(0x7faab80fb150 msgr2=0x7faab8068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.280+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 shutdown_connections 2024-02-19T19:05:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.280+0000 7faabe3e7700 1 -- 172.21.15.39:0/3542066495 wait complete. 2024-02-19T19:05:36.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 -- 172.21.15.39:0/3889377232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 msgr2=0x7f2138103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 --2- 172.21.15.39:0/3889377232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138103b90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f212c009910 tx=0x7f212c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:36.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 -- 172.21.15.39:0/3889377232 shutdown_connections 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 --2- 172.21.15.39:0/3889377232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21381040d0 0x7f21381064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 --2- 172.21.15.39:0/3889377232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138103b90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 -- 172.21.15.39:0/3889377232 >> 172.21.15.39:0/3889377232 conn(0x7f21380fb190 msgr2=0x7f21380fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 -- 172.21.15.39:0/3889377232 shutdown_connections 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 -- 172.21.15.39:0/3889377232 wait complete. 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 Processor -- start 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.385+0000 7f213ee25700 1 -- start start 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213ee25700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213ee25700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21381040d0 0x7f2138198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213ee25700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138199110 con 0x7f21381017a0 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213ee25700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138199250 con 0x7f21381040d0 2024-02-19T19:05:36.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58482/0 (socket says 172.21.15.39:58482) 2024-02-19T19:05:36.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 -- 172.21.15.39:0/2698464673 learned_addr learned my addr 172.21.15.39:0/2698464673 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 -- 172.21.15.39:0/2698464673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21381040d0 msgr2=0x7f2138198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21381040d0 0x7f2138198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 -- 172.21.15.39:0/2698464673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c0095f0 con 0x7f21381017a0 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213cbc1700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138198640 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f212c00bd70 tx=0x7f212c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f212c01d070 con 0x7f21381017a0 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f212c022470 con 0x7f21381017a0 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213819dc60 con 0x7f21381017a0 2024-02-19T19:05:36.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.386+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f212c00f670 con 0x7f21381017a0 2024-02-19T19:05:36.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.387+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213819e120 con 0x7f21381017a0 2024-02-19T19:05:36.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.387+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21380fcd80 con 0x7f21381017a0 2024-02-19T19:05:36.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.388+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7f212c022a50 con 0x7f21381017a0 2024-02-19T19:05:36.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.389+0000 7f2135ffb700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f212806d660 0x7f212806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.389+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f212c08ebd0 con 0x7f21381017a0 2024-02-19T19:05:36.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.389+0000 7f2137fff700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f212806d660 0x7f212806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.389+0000 7f2137fff700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f212806d660 0x7f212806fb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2124005fd0 tx=0x7f2124005d00 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:36.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.393+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f212c05c4b0 con 0x7f21381017a0 2024-02-19T19:05:36.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:36 smithi086 ceph-mon[37629]: pgmap v7: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-19T19:05:36.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:36 smithi086 ceph-mon[37629]: mgrmap e30: smithi039.ydmpuh(active, since 8s), standbys: smithi086.huubbu 2024-02-19T19:05:36.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:36 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T19:05:36.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.690+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f213802cf20 con 0x7f21381017a0 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.691+0000 7f2135ffb700 1 -- 172.21.15.39:0/2698464673 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f212c05c040 con 0x7f21381017a0 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:05:36.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:05:36.712 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:05:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.693+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f212806d660 msgr2=0x7f212806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.693+0000 7f213ee25700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f212806d660 0x7f212806fb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2124005fd0 tx=0x7f2124005d00 comp rx=0 tx=0).stop 2024-02-19T19:05:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.693+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 msgr2=0x7f2138198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.693+0000 7f213ee25700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138198640 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f212c00bd70 tx=0x7f212c00be50 comp rx=0 tx=0).stop 2024-02-19T19:05:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 shutdown_connections 2024-02-19T19:05:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f212806d660 0x7f212806fb20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21381017a0 0x7f2138198640 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 --2- 172.21.15.39:0/2698464673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21381040d0 0x7f2138198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 >> 172.21.15.39:0/2698464673 conn(0x7f21380fb190 msgr2=0x7f2138101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 shutdown_connections 2024-02-19T19:05:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.694+0000 7f213ee25700 1 -- 172.21.15.39:0/2698464673 wait complete. 2024-02-19T19:05:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 -- 172.21.15.39:0/3086127685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 msgr2=0x7fb20c0fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 --2- 172.21.15.39:0/3086127685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c0fffc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb208009910 tx=0x7fb208009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 -- 172.21.15.39:0/3086127685 shutdown_connections 2024-02-19T19:05:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 --2- 172.21.15.39:0/3086127685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb20c100da0 0x7fb20c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 --2- 172.21.15.39:0/3086127685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c0fffc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 -- 172.21.15.39:0/3086127685 >> 172.21.15.39:0/3086127685 conn(0x7fb20c0fb120 msgr2=0x7fb20c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:36.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 -- 172.21.15.39:0/3086127685 shutdown_connections 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.800+0000 7fb21354c700 1 -- 172.21.15.39:0/3086127685 wait complete. 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb21354c700 1 Processor -- start 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb21354c700 1 -- start start 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb21354c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb21354c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb20c100da0 0x7fb20c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb21354c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c199220 con 0x7fb20c0ffba0 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb21354c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c199360 con 0x7fb20c100da0 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54426/0 (socket says 172.21.15.39:54426) 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 -- 172.21.15.39:0/1057895646 learned_addr learned my addr 172.21.15.39:0/1057895646 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb210ae7700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb20c100da0 0x7fb20c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 -- 172.21.15.39:0/1057895646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb20c100da0 msgr2=0x7fb20c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb20c100da0 0x7fb20c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 -- 172.21.15.39:0/1057895646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080095f0 con 0x7fb20c0ffba0 2024-02-19T19:05:36.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.801+0000 7fb2112e8700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c1986c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb20800bda0 tx=0x7fb20800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:36.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.802+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb20801d070 con 0x7fb20c0ffba0 2024-02-19T19:05:36.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.802+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb208004c50 con 0x7fb20c0ffba0 2024-02-19T19:05:36.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.802+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c19ddb0 con 0x7fb20c0ffba0 2024-02-19T19:05:36.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.802+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb20800f690 con 0x7fb20c0ffba0 2024-02-19T19:05:36.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.802+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c19e240 con 0x7fb20c0ffba0 2024-02-19T19:05:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.803+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c108700 con 0x7fb20c0ffba0 2024-02-19T19:05:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.803+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7fb2080229b0 con 0x7fb20c0ffba0 2024-02-19T19:05:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.804+0000 7fb2027fc700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7fb1f806d660 0x7fb1f806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.804+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb20808e8e0 con 0x7fb20c0ffba0 2024-02-19T19:05:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.804+0000 7fb210ae7700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7fb1f806d660 0x7fb1f806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.804+0000 7fb210ae7700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7fb1f806d660 0x7fb1f806fb20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc005fd0 tx=0x7fb1fc00a560 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:36.808+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fb20805c070 con 0x7fb20c0ffba0 2024-02-19T19:05:37.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.066+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb20c19e960 con 0x7fb20c0ffba0 2024-02-19T19:05:37.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.067+0000 7fb2027fc700 1 -- 172.21.15.39:0/1057895646 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1847 (secure 0 0 0) 0x7fb208027750 con 0x7fb20c0ffba0 2024-02-19T19:05:37.087 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:05:37.087 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:05:37.087 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:37.087 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:05:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:05:37.089 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:37.090 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.069+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7fb1f806d660 msgr2=0x7fb1f806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.069+0000 7fb21354c700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7fb1f806d660 0x7fb1f806fb20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc005fd0 tx=0x7fb1fc00a560 comp rx=0 tx=0).stop 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.069+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 msgr2=0x7fb20c1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.069+0000 7fb21354c700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c1986c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb20800bda0 tx=0x7fb20800be80 comp rx=0 tx=0).stop 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.069+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 shutdown_connections 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.070+0000 7fb21354c700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7fb1f806d660 0x7fb1f806fb20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.070+0000 7fb21354c700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb20c0ffba0 0x7fb20c1986c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.070+0000 7fb21354c700 1 --2- 172.21.15.39:0/1057895646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb20c100da0 0x7fb20c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.070+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 >> 172.21.15.39:0/1057895646 conn(0x7fb20c0fb120 msgr2=0x7fb20c068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.070+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 shutdown_connections 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.070+0000 7fb21354c700 1 -- 172.21.15.39:0/1057895646 wait complete. 2024-02-19T19:05:37.091 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:05:37.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.177+0000 7f8637420700 1 -- 172.21.15.39:0/2533465467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 msgr2=0x7f8630100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.177+0000 7f8637420700 1 --2- 172.21.15.39:0/2533465467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630100030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f862c009910 tx=0x7f862c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:37.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.177+0000 7f8637420700 1 -- 172.21.15.39:0/2533465467 shutdown_connections 2024-02-19T19:05:37.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.177+0000 7f8637420700 1 --2- 172.21.15.39:0/2533465467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8630100e10 0x7f8630101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.177+0000 7f8637420700 1 --2- 172.21.15.39:0/2533465467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630100030 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.177+0000 7f8637420700 1 -- 172.21.15.39:0/2533465467 >> 172.21.15.39:0/2533465467 conn(0x7f86300fb190 msgr2=0x7f86300fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:37.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 -- 172.21.15.39:0/2533465467 shutdown_connections 2024-02-19T19:05:37.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 -- 172.21.15.39:0/2533465467 wait complete. 2024-02-19T19:05:37.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 Processor -- start 2024-02-19T19:05:37.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 -- start start 2024-02-19T19:05:37.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8630100e10 0x7f86301021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8630102730 con 0x7f86300ffc10 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f8637420700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8630102870 con 0x7f8630100e10 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.178+0000 7f86351bc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86351bc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54442/0 (socket says 172.21.15.39:54442) 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86351bc700 1 -- 172.21.15.39:0/2567766289 learned_addr learned my addr 172.21.15.39:0/2567766289 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86349bb700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8630100e10 0x7f86301021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86351bc700 1 -- 172.21.15.39:0/2567766289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8630100e10 msgr2=0x7f86301021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86351bc700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8630100e10 0x7f86301021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86351bc700 1 -- 172.21.15.39:0/2567766289 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0095f0 con 0x7f86300ffc10 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86351bc700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630103ba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f862c00bda0 tx=0x7f862c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f862c01d070 con 0x7f86300ffc10 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f862c00f460 con 0x7f86300ffc10 2024-02-19T19:05:37.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8630102af0 con 0x7f86300ffc10 2024-02-19T19:05:37.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f862c021640 con 0x7f86300ffc10 2024-02-19T19:05:37.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.179+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8630102fe0 con 0x7f86300ffc10 2024-02-19T19:05:37.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.180+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8630108700 con 0x7f86300ffc10 2024-02-19T19:05:37.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.181+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7f862c00f5d0 con 0x7f86300ffc10 2024-02-19T19:05:37.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.181+0000 7f86267fc700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f861c06d660 0x7f861c06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:37.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.181+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f862c08e400 con 0x7f86300ffc10 2024-02-19T19:05:37.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.181+0000 7f86349bb700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f861c06d660 0x7f861c06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:37.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.182+0000 7f86349bb700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f861c06d660 0x7f861c06fb20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8620005950 tx=0x7f862000b410 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:37.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.186+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f862c05ba70 con 0x7f86300ffc10 2024-02-19T19:05:37.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.406+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 --> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f863019ea10 con 0x7f861c06d660 2024-02-19T19:05:37.426 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:37 smithi039 ceph-mon[28988]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:37.426 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:37 smithi039 ceph-mon[28988]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:37.426 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:37 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/2698464673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.407+0000 7f86267fc700 1 -- 172.21.15.39:0/2567766289 <== mgr.14744 v2:172.21.15.39:6800/913229036 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f863019ea10 con 0x7f861c06d660 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:05:37.427 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr" 2024-02-19T19:05:37.428 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:05:37.428 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "2/23 daemons upgraded", 2024-02-19T19:05:37.428 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "", 2024-02-19T19:05:37.428 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:05:37.428 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f861c06d660 msgr2=0x7f861c06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f861c06d660 0x7f861c06fb20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8620005950 tx=0x7f862000b410 comp rx=0 tx=0).stop 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 msgr2=0x7f8630103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630103ba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f862c00bda0 tx=0x7f862c00bdd0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 shutdown_connections 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f861c06d660 0x7f861c06fb20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86300ffc10 0x7f8630103ba0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 --2- 172.21.15.39:0/2567766289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8630100e10 0x7f86301021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 >> 172.21.15.39:0/2567766289 conn(0x7f86300fb190 msgr2=0x7f8630068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:37.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 shutdown_connections 2024-02-19T19:05:37.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.409+0000 7f8637420700 1 -- 172.21.15.39:0/2567766289 wait complete. 2024-02-19T19:05:37.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:37 smithi086 ceph-mon[37629]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:37.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:37 smithi086 ceph-mon[37629]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:37.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:37 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2698464673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 -- 172.21.15.39:0/3895464922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 msgr2=0x7f42481064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 --2- 172.21.15.39:0/3895464922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f42481064c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4234009910 tx=0x7f4234009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 -- 172.21.15.39:0/3895464922 shutdown_connections 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 --2- 172.21.15.39:0/3895464922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f42481064c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 --2- 172.21.15.39:0/3895464922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42481017a0 0x7f4248103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 -- 172.21.15.39:0/3895464922 >> 172.21.15.39:0/3895464922 conn(0x7f42480fb190 msgr2=0x7f42480fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 -- 172.21.15.39:0/3895464922 shutdown_connections 2024-02-19T19:05:37.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 -- 172.21.15.39:0/3895464922 wait complete. 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.517+0000 7f424ee6d700 1 Processor -- start 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f424ee6d700 1 -- start start 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f424ee6d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42481017a0 0x7f4248196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f424ee6d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f4248196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f424ee6d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4248196fb0 con 0x7f42481040d0 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f424ee6d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42481970f0 con 0x7f42481017a0 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f4248196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f4248196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54460/0 (socket says 172.21.15.39:54460) 2024-02-19T19:05:37.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 -- 172.21.15.39:0/853785443 learned_addr learned my addr 172.21.15.39:0/853785443 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f424cc09700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42481017a0 0x7f4248196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 -- 172.21.15.39:0/853785443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42481017a0 msgr2=0x7f4248196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42481017a0 0x7f4248196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 -- 172.21.15.39:0/853785443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42340095f0 con 0x7f42481040d0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.518+0000 7f4247fff700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f4248196990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4234006010 tx=0x7f423400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.519+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f423401d070 con 0x7f42481040d0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.519+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424819bb40 con 0x7f42481040d0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.519+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f423400f460 con 0x7f42481040d0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.519+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4234021640 con 0x7f42481040d0 2024-02-19T19:05:37.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.519+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424819c030 con 0x7f42481040d0 2024-02-19T19:05:37.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.520+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4248190610 con 0x7f42481040d0 2024-02-19T19:05:37.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.521+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 30) v1 ==== 91448+0+0 (secure 0 0 0) 0x7f423400fab0 con 0x7f42481040d0 2024-02-19T19:05:37.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.521+0000 7f4245ffb700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f423806d660 0x7f423806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:37.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.521+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f423408e400 con 0x7f42481040d0 2024-02-19T19:05:37.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.521+0000 7f424cc09700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f423806d660 0x7f423806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:37.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.522+0000 7f424cc09700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f423806d660 0x7f423806fb20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f423c005950 tx=0x7f423c00b410 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:37.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.526+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f423405ba70 con 0x7f42481040d0 2024-02-19T19:05:37.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.825+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f424802ce80 con 0x7f42481040d0 2024-02-19T19:05:37.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.826+0000 7f4245ffb700 1 -- 172.21.15.39:0/853785443 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f423400f5d0 con 0x7f42481040d0 2024-02-19T19:05:37.846 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:37.846 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:37.846 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f423806d660 msgr2=0x7f423806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f423806d660 0x7f423806fb20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f423c005950 tx=0x7f423c00b410 comp rx=0 tx=0).stop 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 msgr2=0x7f4248196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f4248196990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4234006010 tx=0x7f423400bd70 comp rx=0 tx=0).stop 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 shutdown_connections 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42481017a0 0x7f4248196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7f423806d660 0x7f423806fb20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 --2- 172.21.15.39:0/853785443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42481040d0 0x7f4248196990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 >> 172.21.15.39:0/853785443 conn(0x7f42480fb190 msgr2=0x7f42480fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 shutdown_connections 2024-02-19T19:05:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:05:37.828+0000 7f424ee6d700 1 -- 172.21.15.39:0/853785443 wait complete. 2024-02-19T19:05:38.387 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:38 smithi039 ceph-mon[28988]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:38.387 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:38 smithi039 ceph-mon[28988]: pgmap v8: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:05:38.387 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:38 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/1057895646' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:38.387 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:38 smithi039 ceph-mon[28988]: from='client.? 172.21.15.39:0/853785443' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:38.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:38 smithi086 ceph-mon[37629]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:38.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:38 smithi086 ceph-mon[37629]: pgmap v8: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:05:38.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:38 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/1057895646' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:38.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:38 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/853785443' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:39.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:39 smithi039 ceph-mon[28988]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:39.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:39 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:39.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:39 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:39.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:39 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:39.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:39 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:39.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:39 smithi086 ceph-mon[37629]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:39.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:39 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:39.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:39 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:39.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:39 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:39.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:39 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: pgmap v9: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:40.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:40.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:40 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: pgmap v9: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:40.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:40 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:41.446 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:41 smithi039 ceph-mon[28988]: Reconfiguring prometheus.smithi039 (dependencies changed)... 2024-02-19T19:05:41.447 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:41 smithi039 ceph-mon[28988]: Reconfiguring daemon prometheus.smithi039 on smithi039 2024-02-19T19:05:41.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:41 smithi086 ceph-mon[37629]: Reconfiguring prometheus.smithi039 (dependencies changed)... 2024-02-19T19:05:41.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:41 smithi086 ceph-mon[37629]: Reconfiguring daemon prometheus.smithi039 on smithi039 2024-02-19T19:05:42.331 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:42 smithi039 ceph-mon[28988]: pgmap v10: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:05:42.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:42 smithi086 ceph-mon[37629]: pgmap v10: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:05:43.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:43 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:43.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:43 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:44.355 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:44 smithi086 ceph-mon[37629]: pgmap v11: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:05:44.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:44 smithi039 ceph-mon[28988]: pgmap v11: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:05:46.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:45 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:46.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:45 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:46.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:45 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:46.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:45 smithi039 ceph-mon[28988]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:46.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:45 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:46.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:45 smithi039 ceph-mon[28988]: pgmap v12: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:05:46.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:45 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:46.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:45 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:46.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:45 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:46.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:45 smithi086 ceph-mon[37629]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:05:46.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:45 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:46.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:45 smithi086 ceph-mon[37629]: pgmap v12: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:05:47.205 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:46 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:05:47.205 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:46 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:05:47.206 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:46 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:47.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:46 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:05:47.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:46 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:05:47.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:46 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:48.219 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:47 smithi086 ceph-mon[37629]: Upgrade: Updating mgr.smithi086.huubbu 2024-02-19T19:05:48.219 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:47 smithi086 ceph-mon[37629]: Deploying daemon mgr.smithi086.huubbu on smithi086 2024-02-19T19:05:48.219 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:47 smithi086 ceph-mon[37629]: pgmap v13: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:48.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:47 smithi039 ceph-mon[28988]: Upgrade: Updating mgr.smithi086.huubbu 2024-02-19T19:05:48.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:47 smithi039 ceph-mon[28988]: Deploying daemon mgr.smithi086.huubbu on smithi086 2024-02-19T19:05:48.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:47 smithi039 ceph-mon[28988]: pgmap v13: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:50.257 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:50 smithi086 ceph-mon[37629]: pgmap v14: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:50.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:50 smithi039 ceph-mon[28988]: pgmap v14: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:51.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:51 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:51.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:51 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:51.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:51 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:51.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:51 smithi039 ceph-mon[28988]: pgmap v15: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:51.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:51 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:51.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:51 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:51.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:51 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:51.770 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:51 smithi086 ceph-mon[37629]: pgmap v15: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:54.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:54 smithi086 ceph-mon[37629]: pgmap v16: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:54.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:54 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:54.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:54 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:54.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:54 smithi039 ceph-mon[28988]: pgmap v16: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:54.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:54 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:54.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:54 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:56.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:55 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:56.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:55 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:56.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:55 smithi039 ceph-mon[28988]: pgmap v17: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:56.284 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:55 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:56.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:55 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:56.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:55 smithi086 ceph-mon[37629]: pgmap v17: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:57.268 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:57 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:57.268 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:57 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:57.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:57 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:57.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:57 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: pgmap v18: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu restarted 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:05:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:58 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: pgmap v18: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: Standby manager daemon smithi086.huubbu restarted 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:05:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:58 smithi039 ceph-mon[28988]: from='mgr.? 172.21.15.86:0/495270738' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:05:59.489 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: mgrmap e31: smithi039.ydmpuh(active, since 31s), standbys: smithi086.huubbu 2024-02-19T19:05:59.489 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi039.ydmpuh"}]: dispatch 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi039.ydmpuh"}]': finished 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.huubbu"}]: dispatch 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.huubbu"}]': finished 2024-02-19T19:05:59.490 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:05:59 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: mgrmap e31: smithi039.ydmpuh(active, since 31s), standbys: smithi086.huubbu 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:59.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:59.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:05:59.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi039.ydmpuh"}]: dispatch 2024-02-19T19:05:59.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi039.ydmpuh"}]': finished 2024-02-19T19:05:59.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.huubbu"}]: dispatch 2024-02-19T19:05:59.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi086.huubbu"}]': finished 2024-02-19T19:05:59.501 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:05:59 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:06:00.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:00 smithi039 ceph-mon[28988]: Upgrade: Setting container_image for all mgr 2024-02-19T19:06:00.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:00 smithi039 ceph-mon[28988]: pgmap v19: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:00.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:00 smithi086 ceph-mon[37629]: Upgrade: Setting container_image for all mgr 2024-02-19T19:06:00.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:00 smithi086 ceph-mon[37629]: pgmap v19: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:01.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: Upgrade: Updating mon.smithi039 2024-02-19T19:06:01.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:01.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:01.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:01.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:01.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: Deploying daemon mon.smithi039 on smithi039 2024-02-19T19:06:01.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:01 smithi039 ceph-mon[28988]: pgmap v20: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: Upgrade: Updating mon.smithi039 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: Deploying daemon mon.smithi039 on smithi039 2024-02-19T19:06:02.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:01 smithi086 ceph-mon[37629]: pgmap v20: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:03.047 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:02 smithi039 systemd[1]: Stopping Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:06:03.431 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[28984]: 2024-02-19T19:06:03.026+0000 7fa1c7e1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi039 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:06:03.432 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[28984]: 2024-02-19T19:06:03.026+0000 7fa1c7e1c700 -1 mon.smithi039@0(leader) e2 *** Got Signal Terminated *** 2024-02-19T19:06:03.432 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:03 smithi039 podman[92051]: 2024-02-19 19:06:03.083763349 +0000 UTC m=+0.184393351 container died ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, io.buildah.version=1.29.1, org.label-schema.build-date=20240214, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-19T19:06:03.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:03 smithi039 podman[92051]: 2024-02-19 19:06:03.602918398 +0000 UTC m=+0.703548399 container cleanup ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.build-date=20240214, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T19:06:03.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:03 smithi039 bash[92051]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039 2024-02-19T19:06:04.083 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:03 smithi039 podman[92077]: 2024-02-19 19:06:03.863585896 +0000 UTC m=+0.780340340 container remove ea8674deb4a5df1c28b6fc338af7b5a83dc629b4530427bb073f3069f7059645 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-19T19:06:04.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service: Deactivated successfully. 2024-02-19T19:06:04.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 systemd[1]: Stopped Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:06:04.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service: Consumed 11.643s CPU time. 2024-02-19T19:06:04.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 systemd[1]: Starting Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:06:04.796 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 podman[92205]: 2024-02-19 19:06:04.402291504 +0000 UTC m=+0.017981984 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:04.796 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 podman[92205]: 2024-02-19 19:06:04.548581419 +0000 UTC m=+0.164271937 container create 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-19T19:06:04.796 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 podman[92205]: 2024-02-19 19:06:04.776170323 +0000 UTC m=+0.391860796 container init 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-19T19:06:05.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 podman[92205]: 2024-02-19 19:06:04.77929817 +0000 UTC m=+0.394988642 container start 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev), process ceph-mon, pid 2 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: pidfile_write: ignore empty --pid-file 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: load: jerasure load: lrc 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: RocksDB version: 7.9.2 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Git sha 0 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Compile date 2024-02-15 18:05:23 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: DB SUMMARY 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: DB Session ID: ARJDSLK6R06USRBRYLD8 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: CURRENT file: CURRENT 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1509 Bytes 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi039/store.db dir, Total Num: 1, files: 000032.sst 2024-02-19T19:06:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi039/store.db: 000030.log size: 1199614 ; 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.error_if_exists: 0 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.create_if_missing: 0 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.paranoid_checks: 1 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.env: 0x5564b7a4cae0 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.info_log: 0x5564b87ab760 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.statistics: (nil) 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.use_fsync: 0 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_log_file_size: 0 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T19:06:05.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T19:06:05.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T19:06:05.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T19:06:05.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.allow_fallocate: 1 2024-02-19T19:06:05.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T19:06:05.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T19:06:05.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.use_direct_reads: 0 2024-02-19T19:06:05.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.db_log_dir: 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.wal_dir: 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T19:06:05.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.write_buffer_manager: 0x5564b877f400 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.unordered_write: 0 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.row_cache: None 2024-02-19T19:06:05.253 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.wal_filter: None 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.two_write_queues: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.wal_compression: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.atomic_flush: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.log_readahead_size: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T19:06:05.254 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_background_jobs: 2 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_background_compactions: -1 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_subcompactions: 1 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T19:06:05.255 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_open_files: -1 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_background_flushes: -1 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Compression algorithms supported: 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kZSTD supported: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kXpressCompression supported: 0 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kZlibCompression supported: 1 2024-02-19T19:06:05.256 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kSnappyCompression supported: 1 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kLZ4Compression supported: 1 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: kBZip2Compression supported: 0 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi039/store.db/MANIFEST-000015 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.merge_operator: 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_filter: None 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T19:06:05.257 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5564b87ab880) 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: cache_index_and_filter_blocks: 1 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: pin_top_level_index_and_filter: 1 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: index_type: 0 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: data_block_index_type: 0 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: index_shortening: 1 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: checksum: 4 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: no_block_cache: 0 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache: 0x5564b87af090 2024-02-19T19:06:05.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache_name: BinnedLRUCache 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache_options: 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: capacity : 536870912 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: num_shard_bits : 4 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: strict_capacity_limit : 0 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: high_pri_pool_ratio: 0.000 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_cache_compressed: (nil) 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: persistent_cache: (nil) 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_size: 4096 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_size_deviation: 10 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_restart_interval: 16 2024-02-19T19:06:05.259 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: index_block_restart_interval: 1 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: metadata_block_size: 4096 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: partition_filters: 0 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: use_delta_encoding: 1 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: filter_policy: bloomfilter 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: whole_key_filtering: 1 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: verify_compression: 0 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: read_amp_bytes_per_bit: 0 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: format_version: 5 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: enable_index_compression: 1 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: block_align: 0 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: max_auto_readahead_size: 262144 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: prepopulate_block_cache: 0 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: initial_auto_readahead_size: 8192 2024-02-19T19:06:05.260 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression: NoCompression 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.num_levels: 7 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T19:06:05.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:06:05.262 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T19:06:05.263 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T19:06:05.264 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.table_properties_collectors: 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.inplace_update_support: 0 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.bloom_locality: 0 2024-02-19T19:06:05.265 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.max_successive_merges: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.ttl: 2592000 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.enable_blob_files: false 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.min_blob_size: 0 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T19:06:05.266 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi039/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12090, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fa20b246-dfe6-4930-93e3-91065e90de6b 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369564818499, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369564822667, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 1076844, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12091, "largest_seqno": 12697, "table_properties": {"data_size": 1073409, "index_size": 1717, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 6545, "raw_average_key_size": 23, "raw_value_size": 1067466, "raw_average_value_size": 3910, "num_data_blocks": 78, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708369564, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fa20b246-dfe6-4930-93e3-91065e90de6b", "db_session_id": "ARJDSLK6R06USRBRYLD8", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369564822759, "job": 1, "event": "recovery_finished"} 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:06:05.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi039/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5564b889e000 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: DB pointer 0x5564b888a000 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** DB Stats ** 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:06:05.268 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** Compaction Stats [default] ** 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: L0 1/0 1.03 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 432.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: L6 1/0 5.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Sum 2/0 6.65 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 432.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 432.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** Compaction Stats [default] ** 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 432.6 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.269 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Cumulative compaction: 0.00 GB write, 60.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Interval compaction: 0.00 GB write, 60.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Block cache BinnedLRUCache@0x5564b87af090#2 capacity: 512.00 MB usage: 2.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.78 KB,0.000149012%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.270 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: starting mon.smithi039 rank 0 at public addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] at bind addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi039 fsid 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???) e2 preinit fsid 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).mds e15 new map 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).mds e15 print_map 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: e15 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: legacy client fscid: 1 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Filesystem 'cephfs' (1) 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: fs_name cephfs 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: epoch 15 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:06:05.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: tableserver 0 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: root 0 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: session_timeout 60 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: session_autoclose 300 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: max_file_size 1099511627776 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: max_xattr_size 65536 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: required_client_features {} 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: last_failure 0 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: last_failure_osd_epoch 46 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: max_mds 1 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: in 0 2024-02-19T19:06:05.272 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: up {0=24267} 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: failed 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: damaged 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: stopped 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: data_pools [3] 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: metadata_pool 2 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: inline_data enabled 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: balancer 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: bal_rank_mask -1 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: standby_count_wanted 1 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: [mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: [mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.273 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: Standby daemons: 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: [mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout: [mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 ceph-mon[92224]: mon.smithi039@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 bash[92205]: 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 2024-02-19T19:06:05.274 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:04 smithi039 systemd[1]: Started Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:06:06.543 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: pgmap v21: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: pgmap v22: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: mon.smithi039 calling monitor election 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: mon.smithi039 is new leader, mons smithi039,smithi086 in quorum (ranks 0,1) 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: monmap e2: 2 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: mgrmap e31: smithi039.ydmpuh(active, since 39s), standbys: smithi086.huubbu 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:06.544 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:06.545 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:06.545 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: from='mgr.14744 ' entity='' 2024-02-19T19:06:06.545 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:06 smithi039 ceph-mon[92224]: mgrmap e32: smithi039.ydmpuh(active, since 39s), standbys: smithi086.huubbu 2024-02-19T19:06:06.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: pgmap v21: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:06.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:06:06.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: from='mgr.14744 172.21.15.39:0/3697364387' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: pgmap v22: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: mon.smithi039 calling monitor election 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: mon.smithi039 is new leader, mons smithi039,smithi086 in quorum (ranks 0,1) 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: monmap e2: 2 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:06:06.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: mgrmap e31: smithi039.ydmpuh(active, since 39s), standbys: smithi086.huubbu 2024-02-19T19:06:06.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:06.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:06.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:06.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: from='mgr.14744 ' entity='' 2024-02-19T19:06:06.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:06 smithi086 ceph-mon[37629]: mgrmap e32: smithi039.ydmpuh(active, since 39s), standbys: smithi086.huubbu 2024-02-19T19:06:07.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 -- 172.21.15.39:0/2352481885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634103e60 msgr2=0x7ff634106250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:07.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 --2- 172.21.15.39:0/2352481885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634103e60 0x7ff634106250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff624009910 tx=0x7ff624009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 -- 172.21.15.39:0/2352481885 shutdown_connections 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 --2- 172.21.15.39:0/2352481885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634103e60 0x7ff634106250 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 --2- 172.21.15.39:0/2352481885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634101530 0x7ff634103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 -- 172.21.15.39:0/2352481885 >> 172.21.15.39:0/2352481885 conn(0x7ff6340faf40 msgr2=0x7ff6340fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 -- 172.21.15.39:0/2352481885 shutdown_connections 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.940+0000 7ff638f9d700 1 -- 172.21.15.39:0/2352481885 wait complete. 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff638f9d700 1 Processor -- start 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff638f9d700 1 -- start start 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff638f9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 0x7ff634196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff638f9d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634103e60 0x7ff634196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff638f9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff634196db0 con 0x7ff634101530 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff638f9d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff634196ef0 con 0x7ff634103e60 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff6337fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 0x7ff634196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff6337fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 0x7ff634196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41140/0 (socket says 172.21.15.39:41140) 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff6337fe700 1 -- 172.21.15.39:0/873141383 learned_addr learned my addr 172.21.15.39:0/873141383 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634103e60 0x7ff634196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.941+0000 7ff6337fe700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634103e60 msgr2=0x7ff634196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff6337fe700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634103e60 0x7ff634196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff6337fe700 1 -- 172.21.15.39:0/873141383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240095f0 con 0x7ff634101530 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634103e60 0x7ff634196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff6337fe700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 0x7ff634196250 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff62800e920 tx=0x7ff62800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:07.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62800c9d0 con 0x7ff634101530 2024-02-19T19:06:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63419b9a0 con 0x7ff634101530 2024-02-19T19:06:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff628010430 con 0x7ff634101530 2024-02-19T19:06:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff628018650 con 0x7ff634101530 2024-02-19T19:06:07.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.942+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63419bef0 con 0x7ff634101530 2024-02-19T19:06:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.943+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff634190410 con 0x7ff634101530 2024-02-19T19:06:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.944+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 32) v1 ==== 91595+0+0 (secure 0 0 0) 0x7ff6280188c0 con 0x7ff634101530 2024-02-19T19:06:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.944+0000 7ff630ff9700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.944+0000 7ff632ffd700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 msgr2=0x7ff61c06fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/913229036 2024-02-19T19:06:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.944+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff628014070 con 0x7ff634101530 2024-02-19T19:06:07.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.944+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T19:06:07.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:07.949+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff628052d20 con 0x7ff634101530 2024-02-19T19:06:08.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:08.145+0000 7ff632ffd700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 msgr2=0x7ff61c06fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/913229036 2024-02-19T19:06:08.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:08.145+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T19:06:08.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:08.172+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 --> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6340611d0 con 0x7ff61c06d9e0 2024-02-19T19:06:08.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:08.546+0000 7ff632ffd700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 msgr2=0x7ff61c06fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/913229036 2024-02-19T19:06:08.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:08.546+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T19:06:09.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:09.347+0000 7ff632ffd700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 msgr2=0x7ff61c06fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/913229036 2024-02-19T19:06:09.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:09.347+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T19:06:10.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:10.948+0000 7ff632ffd700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 msgr2=0x7ff61c06fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/913229036 2024-02-19T19:06:10.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:10.948+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T19:06:13.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:13.211+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 33) v1 ==== 91595+0+0 (secure 0 0 0) 0x7ff62808bb70 con 0x7ff634101530 2024-02-19T19:06:13.413 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:13 smithi039 ceph-mon[92224]: Standby manager daemon smithi086.huubbu restarted 2024-02-19T19:06:13.414 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:13 smithi039 ceph-mon[92224]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:06:13.414 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:13 smithi039 ceph-mon[92224]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:06:13.414 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:13 smithi039 ceph-mon[92224]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:06:13.414 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:13 smithi039 ceph-mon[92224]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:06:13.414 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:13 smithi039 ceph-mon[92224]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:06:13.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:13.412+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mgrmap(e 34) v1 ==== 91118+0+0 (secure 0 0 0) 0x7ff628057410 con 0x7ff634101530 2024-02-19T19:06:13.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:13.412+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 msgr2=0x7ff61c06fea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T19:06:13.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:13.412+0000 7ff630ff9700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:13 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu restarted 2024-02-19T19:06:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:13 smithi086 ceph-mon[37629]: Standby manager daemon smithi086.huubbu started 2024-02-19T19:06:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:13 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/crt"}]: dispatch 2024-02-19T19:06:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:13 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:06:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:13 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi086.huubbu/key"}]: dispatch 2024-02-19T19:06:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:13 smithi086 ceph-mon[37629]: from='mgr.? 172.21.15.86:0/1596806141' entity='mgr.smithi086.huubbu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:06:14.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.414+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mon.0 v2:172.21.15.39:3300/0 9 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7ff628057ce0 con 0x7ff634101530 2024-02-19T19:06:14.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.414+0000 7ff630ff9700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff61c028ee0 0x7ff61c077400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.414+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6340611d0 con 0x7ff61c028ee0 2024-02-19T19:06:14.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.416+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff61c028ee0 0x7ff61c077400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.416+0000 7ff632ffd700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff61c028ee0 0x7ff61c077400 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff624009fd0 tx=0x7ff624005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:14.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.419+0000 7ff630ff9700 1 -- 172.21.15.39:0/873141383 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7ff6340611d0 con 0x7ff61c028ee0 2024-02-19T19:06:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff61c028ee0 msgr2=0x7ff61c077400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff61c028ee0 0x7ff61c077400 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff624009fd0 tx=0x7ff624005940 comp rx=0 tx=0).stop 2024-02-19T19:06:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 msgr2=0x7ff634196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 0x7ff634196250 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff62800e920 tx=0x7ff62800ec30 comp rx=0 tx=0).stop 2024-02-19T19:06:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 shutdown_connections 2024-02-19T19:06:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff61c028ee0 0x7ff61c077400 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:6800/913229036,v1:172.21.15.39:6801/913229036] conn(0x7ff61c06d9e0 0x7ff61c06fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff634101530 0x7ff634196250 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 --2- 172.21.15.39:0/873141383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff634103e60 0x7ff634196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 >> 172.21.15.39:0/873141383 conn(0x7ff6340faf40 msgr2=0x7ff6340fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:14.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 shutdown_connections 2024-02-19T19:06:14.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.423+0000 7ff638f9d700 1 -- 172.21.15.39:0/873141383 wait complete. 2024-02-19T19:06:14.457 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: mgrmap e33: smithi039.ydmpuh(active, since 46s), standbys: smithi086.huubbu 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: Active manager daemon smithi039.ydmpuh restarted 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: Activating manager daemon smithi039.ydmpuh 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: mgrmap e34: smithi039.ydmpuh(active, starting, since 0.00753589s), standbys: smithi086.huubbu 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:06:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:14.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:14.500 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:14.500 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:06:14.500 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: mgrmap e33: smithi039.ydmpuh(active, since 46s), standbys: smithi086.huubbu 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: Active manager daemon smithi039.ydmpuh restarted 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: Activating manager daemon smithi039.ydmpuh 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: mgrmap e34: smithi039.ydmpuh(active, starting, since 0.00753589s), standbys: smithi086.huubbu 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:06:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi039.ydmpuh", "id": "smithi039.ydmpuh"}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr metadata", "who": "smithi086.huubbu", "id": "smithi086.huubbu"}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: Manager daemon smithi039.ydmpuh is now available 2024-02-19T19:06:14.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:14.538 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:14.538 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:14.538 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:06:14.538 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:14 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi039.ydmpuh/trash_purge_schedule"}]: dispatch 2024-02-19T19:06:14.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 -- 172.21.15.39:0/3629603813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 msgr2=0x7efda0101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 --2- 172.21.15.39:0/3629603813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda0101270 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd94009910 tx=0x7efd94009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:14.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 -- 172.21.15.39:0/3629603813 shutdown_connections 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 --2- 172.21.15.39:0/3629603813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda0101270 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 --2- 172.21.15.39:0/3629603813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda00ffc10 0x7efda0100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 -- 172.21.15.39:0/3629603813 >> 172.21.15.39:0/3629603813 conn(0x7efda00fb190 msgr2=0x7efda00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 -- 172.21.15.39:0/3629603813 shutdown_connections 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.534+0000 7efda5e3d700 1 -- 172.21.15.39:0/3629603813 wait complete. 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.535+0000 7efda5e3d700 1 Processor -- start 2024-02-19T19:06:14.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.535+0000 7efda5e3d700 1 -- start start 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.535+0000 7efda5e3d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda00ffc10 0x7efda00690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.535+0000 7efda5e3d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.535+0000 7efda5e3d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda019b7d0 con 0x7efda0100e10 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.535+0000 7efda5e3d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda019b910 con 0x7efda00ffc10 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda019b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41242/0 (socket says 172.21.15.39:41242) 2024-02-19T19:06:14.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 -- 172.21.15.39:0/2781786256 learned_addr learned my addr 172.21.15.39:0/2781786256 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:14.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9f7fe700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda00ffc10 0x7efda00690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 -- 172.21.15.39:0/2781786256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda00ffc10 msgr2=0x7efda00690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda00ffc10 0x7efda00690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 -- 172.21.15.39:0/2781786256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940095f0 con 0x7efda0100e10 2024-02-19T19:06:14.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9effd700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda019b200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd9400fa00 tx=0x7efd9400fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:14.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd9401d070 con 0x7efda0100e10 2024-02-19T19:06:14.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efd94005640 con 0x7efda0100e10 2024-02-19T19:06:14.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda019bb90 con 0x7efda0100e10 2024-02-19T19:06:14.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd94017630 con 0x7efda0100e10 2024-02-19T19:06:14.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.536+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda019c080 con 0x7efda0100e10 2024-02-19T19:06:14.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.537+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda0108700 con 0x7efda0100e10 2024-02-19T19:06:14.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.538+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7efd94003680 con 0x7efda0100e10 2024-02-19T19:06:14.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.538+0000 7efd9cff9700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efd8806d5c0 0x7efd8806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.538+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efd9408e720 con 0x7efda0100e10 2024-02-19T19:06:14.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.538+0000 7efd9f7fe700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efd8806d5c0 0x7efd8806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.539+0000 7efd9f7fe700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efd8806d5c0 0x7efd8806fa80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efda0100c70 tx=0x7efd90009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:14.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.543+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd94058100 con 0x7efda0100e10 2024-02-19T19:06:14.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.766+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efda0105760 con 0x7efd8806d5c0 2024-02-19T19:06:14.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.767+0000 7efd9cff9700 1 -- 172.21.15.39:0/2781786256 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7efda0105760 con 0x7efd8806d5c0 2024-02-19T19:06:14.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efd8806d5c0 msgr2=0x7efd8806fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efd8806d5c0 0x7efd8806fa80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efda0100c70 tx=0x7efd90009260 comp rx=0 tx=0).stop 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 msgr2=0x7efda019b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda019b200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efd9400fa00 tx=0x7efd9400fa30 comp rx=0 tx=0).stop 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 shutdown_connections 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efd8806d5c0 0x7efd8806fa80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda00ffc10 0x7efda00690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 --2- 172.21.15.39:0/2781786256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efda0100e10 0x7efda019b200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 >> 172.21.15.39:0/2781786256 conn(0x7efda00fb190 msgr2=0x7efda0104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 shutdown_connections 2024-02-19T19:06:14.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.769+0000 7efda5e3d700 1 -- 172.21.15.39:0/2781786256 wait complete. 2024-02-19T19:06:14.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 -- 172.21.15.39:0/317502878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 msgr2=0x7fc39c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 --2- 172.21.15.39:0/317502878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c101270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc390009910 tx=0x7fc390009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:14.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 -- 172.21.15.39:0/317502878 shutdown_connections 2024-02-19T19:06:14.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 --2- 172.21.15.39:0/317502878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c101270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 --2- 172.21.15.39:0/317502878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc39c0ffc10 0x7fc39c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 -- 172.21.15.39:0/317502878 >> 172.21.15.39:0/317502878 conn(0x7fc39c0fb190 msgr2=0x7fc39c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 -- 172.21.15.39:0/317502878 shutdown_connections 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.879+0000 7fc3a1475700 1 -- 172.21.15.39:0/317502878 wait complete. 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc3a1475700 1 Processor -- start 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc3a1475700 1 -- start start 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc3a1475700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc39c0ffc10 0x7fc39c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc3a1475700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc3a1475700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c196fe0 con 0x7fc39c100e10 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc3a1475700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c197120 con 0x7fc39c0ffc10 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc39a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc39a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c1969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41260/0 (socket says 172.21.15.39:41260) 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc39a7fc700 1 -- 172.21.15.39:0/3497375613 learned_addr learned my addr 172.21.15.39:0/3497375613 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:14.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.880+0000 7fc39affd700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc39c0ffc10 0x7fc39c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc39a7fc700 1 -- 172.21.15.39:0/3497375613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc39c0ffc10 msgr2=0x7fc39c196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc39a7fc700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc39c0ffc10 0x7fc39c196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc39a7fc700 1 -- 172.21.15.39:0/3497375613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900095f0 con 0x7fc39c100e10 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc39a7fc700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c1969c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc390006010 tx=0x7fc39000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39001d070 con 0x7fc39c100e10 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39c19bb70 con 0x7fc39c100e10 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc39000f460 con 0x7fc39c100e10 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc390021640 con 0x7fc39c100e10 2024-02-19T19:06:14.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.881+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c19c060 con 0x7fc39c100e10 2024-02-19T19:06:14.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.882+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39c108700 con 0x7fc39c100e10 2024-02-19T19:06:14.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.883+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7fc39000fab0 con 0x7fc39c100e10 2024-02-19T19:06:14.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.883+0000 7fc383fff700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc38406d5c0 0x7fc38406fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:14.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.884+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc39008e1b0 con 0x7fc39c100e10 2024-02-19T19:06:14.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.884+0000 7fc39affd700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc38406d5c0 0x7fc38406fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:14.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.884+0000 7fc39affd700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc38406d5c0 0x7fc38406fa80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc39c100c70 tx=0x7fc38c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:14.888+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc390057c90 con 0x7fc39c100e10 2024-02-19T19:06:15.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.106+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc39c068c90 con 0x7fc38406d5c0 2024-02-19T19:06:15.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.118+0000 7fc383fff700 1 -- 172.21.15.39:0/3497375613 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc39c068c90 con 0x7fc38406d5c0 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (8m) 19s ago 9m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (10m) 19s ago 10m 7734k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (8m) 21s ago 8m 7658k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (10m) 19s ago 10m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (8m) 21s ago 8m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (8m) 19s ago 9m 86.2M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (4m) 19s ago 4m 24.1M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (4m) 19s ago 4m 13.9M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:06:15.139 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (4m) 21s ago 4m 28.9M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (4m) 21s ago 4m 15.6M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (59s) 19s ago 11m 493M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (25s) 21s ago 8m 68.7M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (11m) 19s ago 11m 54.0M 2048M 18.2.1 94eadd990e0a ea8674deb4a5 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (8m) 21s ago 8m 40.5M 2048M 18.2.1 94eadd990e0a 99602b1b0a01 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (10m) 19s ago 10m 14.4M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (8m) 21s ago 8m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (7m) 19s ago 7m 51.5M 1216M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (7m) 19s ago 7m 50.3M 1216M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (6m) 19s ago 6m 48.9M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:06:15.140 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (6m) 21s ago 6m 48.7M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:06:15.141 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (6m) 21s ago 6m 46.6M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:06:15.141 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (5m) 21s ago 5m 48.6M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:06:15.141 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (30s) 19s ago 9m 37.2M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc38406d5c0 msgr2=0x7fc38406fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc38406d5c0 0x7fc38406fa80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc39c100c70 tx=0x7fc38c00b410 comp rx=0 tx=0).stop 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 msgr2=0x7fc39c1969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c1969c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc390006010 tx=0x7fc39000bd70 comp rx=0 tx=0).stop 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 shutdown_connections 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc38406d5c0 0x7fc38406fa80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc39c0ffc10 0x7fc39c196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 --2- 172.21.15.39:0/3497375613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc39c100e10 0x7fc39c1969c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 >> 172.21.15.39:0/3497375613 conn(0x7fc39c0fb190 msgr2=0x7fc39c068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 shutdown_connections 2024-02-19T19:06:15.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.120+0000 7fc3a1475700 1 -- 172.21.15.39:0/3497375613 wait complete. 2024-02-19T19:06:15.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 -- 172.21.15.39:0/2882968288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d40fed20 msgr2=0x7f69d40ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 --2- 172.21.15.39:0/2882968288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d40fed20 0x7f69d40ff1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009910 tx=0x7f69c4009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:15.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 -- 172.21.15.39:0/2882968288 shutdown_connections 2024-02-19T19:06:15.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 --2- 172.21.15.39:0/2882968288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d40fed20 0x7f69d40ff1a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 --2- 172.21.15.39:0/2882968288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d41022a0 0x7f69d41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 -- 172.21.15.39:0/2882968288 >> 172.21.15.39:0/2882968288 conn(0x7f69d40fa8c0 msgr2=0x7f69d40fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:15.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 -- 172.21.15.39:0/2882968288 shutdown_connections 2024-02-19T19:06:15.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.225+0000 7f69db9cd700 1 -- 172.21.15.39:0/2882968288 wait complete. 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69db9cd700 1 Processor -- start 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69db9cd700 1 -- start start 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69db9cd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d40fed20 0x7f69d41964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69db9cd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 0x7f69d4196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69db9cd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4197030 con 0x7f69d41022a0 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69db9cd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d4197170 con 0x7f69d40fed20 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69d8f68700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 0x7f69d4196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69d8f68700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 0x7f69d4196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41286/0 (socket says 172.21.15.39:41286) 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.226+0000 7f69d8f68700 1 -- 172.21.15.39:0/4006103647 learned_addr learned my addr 172.21.15.39:0/4006103647 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69d9769700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d40fed20 0x7f69d41964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:15.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69d8f68700 1 -- 172.21.15.39:0/4006103647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d40fed20 msgr2=0x7f69d41964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69d8f68700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d40fed20 0x7f69d41964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69d8f68700 1 -- 172.21.15.39:0/4006103647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40095f0 con 0x7f69d41022a0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69d8f68700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 0x7f69d4196a10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69c4000c00 tx=0x7f69c400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c401d070 con 0x7f69d41022a0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f69c4005dc0 con 0x7f69d41022a0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d419bbc0 con 0x7f69d41022a0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69c4022470 con 0x7f69d41022a0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.227+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d419c0b0 con 0x7f69d41022a0 2024-02-19T19:06:15.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.228+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d4109820 con 0x7f69d41022a0 2024-02-19T19:06:15.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.229+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7f69c4003810 con 0x7f69d41022a0 2024-02-19T19:06:15.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.229+0000 7f69ca7fc700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f69c006d5c0 0x7f69c006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:15.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.229+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f69c408e430 con 0x7f69d41022a0 2024-02-19T19:06:15.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.229+0000 7f69d9769700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f69c006d5c0 0x7f69c006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:15.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.229+0000 7f69d9769700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f69c006d5c0 0x7f69c006fa80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69d4068b10 tx=0x7f69d0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:15.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.234+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f69c4057f10 con 0x7f69d41022a0 2024-02-19T19:06:15.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:15 smithi039 ceph-mon[92224]: [19/Feb/2024:19:06:14] ENGINE Bus STARTING 2024-02-19T19:06:15.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:15 smithi039 ceph-mon[92224]: mgrmap e35: smithi039.ydmpuh(active, since 1.01135s), standbys: smithi086.huubbu 2024-02-19T19:06:15.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:15 smithi086 ceph-mon[37629]: [19/Feb/2024:19:06:14] ENGINE Bus STARTING 2024-02-19T19:06:15.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:15 smithi086 ceph-mon[37629]: mgrmap e35: smithi039.ydmpuh(active, since 1.01135s), standbys: smithi086.huubbu 2024-02-19T19:06:15.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.575+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f69d419c7d0 con 0x7f69d41022a0 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.576+0000 7f69ca7fc700 1 -- 172.21.15.39:0/4006103647 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+779 (secure 0 0 0) 0x7f69c405b5d0 con 0x7f69d41022a0 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 11, 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 3 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:06:15.598 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:06:15.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f69c006d5c0 msgr2=0x7f69c006fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f69c006d5c0 0x7f69c006fa80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69d4068b10 tx=0x7f69d0009260 comp rx=0 tx=0).stop 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 msgr2=0x7f69d4196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 0x7f69d4196a10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f69c4000c00 tx=0x7f69c400f690 comp rx=0 tx=0).stop 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 shutdown_connections 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f69c006d5c0 0x7f69c006fa80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d40fed20 0x7f69d41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 --2- 172.21.15.39:0/4006103647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69d41022a0 0x7f69d4196a10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.578+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 >> 172.21.15.39:0/4006103647 conn(0x7f69d40fa8c0 msgr2=0x7f69d4105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.579+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 shutdown_connections 2024-02-19T19:06:15.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.579+0000 7f69db9cd700 1 -- 172.21.15.39:0/4006103647 wait complete. 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- 172.21.15.39:0/918148859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e00ff320 msgr2=0x7f85e00ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 --2- 172.21.15.39:0/918148859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e00ff320 0x7f85e00ff740 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009910 tx=0x7f85d8009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- 172.21.15.39:0/918148859 shutdown_connections 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 --2- 172.21.15.39:0/918148859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e0100520 0x7f85e0100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 --2- 172.21.15.39:0/918148859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e00ff320 0x7f85e00ff740 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- 172.21.15.39:0/918148859 >> 172.21.15.39:0/918148859 conn(0x7f85e00fa8c0 msgr2=0x7f85e00fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- 172.21.15.39:0/918148859 shutdown_connections 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- 172.21.15.39:0/918148859 wait complete. 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 Processor -- start 2024-02-19T19:06:15.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- start start 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e00ff320 0x7f85e0106320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 0x7f85e019b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e019b7f0 con 0x7f85e0100520 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.684+0000 7f85e83d2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e019b930 con 0x7f85e00ff320 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 0x7f85e019b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 0x7f85e019b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41334/0 (socket says 172.21.15.39:41334) 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 -- 172.21.15.39:0/2312906055 learned_addr learned my addr 172.21.15.39:0/2312906055 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e616e700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e00ff320 0x7f85e0106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 -- 172.21.15.39:0/2312906055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e00ff320 msgr2=0x7f85e0106320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e00ff320 0x7f85e0106320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 -- 172.21.15.39:0/2312906055 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80095f0 con 0x7f85e0100520 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e616e700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e00ff320 0x7f85e0106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e596d700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 0x7f85e019b1d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f85d400c8a0 tx=0x7f85d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85d400cea0 con 0x7f85e0100520 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f85d4004830 con 0x7f85e0100520 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e019bc10 con 0x7f85e0100520 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85d40055b0 con 0x7f85e0100520 2024-02-19T19:06:15.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.685+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e019c160 con 0x7f85e0100520 2024-02-19T19:06:15.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.686+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0108700 con 0x7f85e0100520 2024-02-19T19:06:15.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.687+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7f85d4007ad0 con 0x7f85e0100520 2024-02-19T19:06:15.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.687+0000 7f85df7fe700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f85c806d5c0 0x7f85c806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:15.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.687+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f85d408cdd0 con 0x7f85e0100520 2024-02-19T19:06:15.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.687+0000 7f85e616e700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f85c806d5c0 0x7f85c806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:15.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.688+0000 7f85e616e700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f85c806d5c0 0x7f85c806fa80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f85d8006010 tx=0x7f85d800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:15.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.692+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85d40567b0 con 0x7f85e0100520 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.946+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f85e019c7d0 con 0x7f85e0100520 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.946+0000 7f85df7fe700 1 -- 172.21.15.39:0/2312906055 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f85d401d070 con 0x7f85e0100520 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:06:15.967 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:06:15.968 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:06:15.969 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:15.970 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f85c806d5c0 msgr2=0x7f85c806fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f85c806d5c0 0x7f85c806fa80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f85d8006010 tx=0x7f85d800b540 comp rx=0 tx=0).stop 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 msgr2=0x7f85e019b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 0x7f85e019b1d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f85d400c8a0 tx=0x7f85d400cc60 comp rx=0 tx=0).stop 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 shutdown_connections 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f85c806d5c0 0x7f85c806fa80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85e00ff320 0x7f85e0106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 --2- 172.21.15.39:0/2312906055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85e0100520 0x7f85e019b1d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:15.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 >> 172.21.15.39:0/2312906055 conn(0x7f85e00fa8c0 msgr2=0x7f85e0103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:15.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.948+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 shutdown_connections 2024-02-19T19:06:15.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:15.949+0000 7f85e83d2700 1 -- 172.21.15.39:0/2312906055 wait complete. 2024-02-19T19:06:15.972 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:06:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/1510323253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 msgr2=0x7ff4b8101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/1510323253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b8101220 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009910 tx=0x7ff4a0009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/1510323253 shutdown_connections 2024-02-19T19:06:16.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/1510323253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b8101220 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/1510323253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4b80ffbc0 0x7ff4b80fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/1510323253 >> 172.21.15.39:0/1510323253 conn(0x7ff4b80fb160 msgr2=0x7ff4b80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/1510323253 shutdown_connections 2024-02-19T19:06:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.056+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/1510323253 wait complete. 2024-02-19T19:06:16.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4bd2f4700 1 Processor -- start 2024-02-19T19:06:16.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4bd2f4700 1 -- start start 2024-02-19T19:06:16.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4bd2f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4b80ffbc0 0x7ff4b8103b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4bd2f4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b81021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4bd2f4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b81026e0 con 0x7ff4b8100dc0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4bd2f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b8102820 con 0x7ff4b80ffbc0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b81021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b81021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41348/0 (socket says 172.21.15.39:41348) 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b67fc700 1 -- 172.21.15.39:0/2007681858 learned_addr learned my addr 172.21.15.39:0/2007681858 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b6ffd700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4b80ffbc0 0x7ff4b8103b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b67fc700 1 -- 172.21.15.39:0/2007681858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4b80ffbc0 msgr2=0x7ff4b8103b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b67fc700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4b80ffbc0 0x7ff4b8103b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.057+0000 7ff4b67fc700 1 -- 172.21.15.39:0/2007681858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00095f0 con 0x7ff4b8100dc0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.058+0000 7ff4b67fc700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b81021a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000bd70 tx=0x7ff4a000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.058+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a001d070 con 0x7ff4b8100dc0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.058+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff4a0005dc0 con 0x7ff4b8100dc0 2024-02-19T19:06:16.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.058+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4a0021410 con 0x7ff4b8100dc0 2024-02-19T19:06:16.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.058+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b8102aa0 con 0x7ff4b8100dc0 2024-02-19T19:06:16.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.058+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b8102f40 con 0x7ff4b8100dc0 2024-02-19T19:06:16.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.059+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b8068af0 con 0x7ff4b8100dc0 2024-02-19T19:06:16.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.059+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7ff4a000f7d0 con 0x7ff4b8100dc0 2024-02-19T19:06:16.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.059+0000 7ff4affff700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff4a406d5c0 0x7ff4a406fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:16.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.060+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff4a008e320 con 0x7ff4b8100dc0 2024-02-19T19:06:16.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.060+0000 7ff4b6ffd700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff4a406d5c0 0x7ff4a406fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:16.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.060+0000 7ff4b6ffd700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff4a406d5c0 0x7ff4a406fa80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8100c20 tx=0x7ff4a8007400 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:16.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.064+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4a0057d80 con 0x7ff4b8100dc0 2024-02-19T19:06:16.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.293+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4b8069560 con 0x7ff4a406d5c0 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.295+0000 7ff4affff700 1 -- 172.21.15.39:0/2007681858 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7ff4b8069560 con 0x7ff4a406d5c0 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr" 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:06:16.316 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "2/23 daemons upgraded", 2024-02-19T19:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "", 2024-02-19T19:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:06:16.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff4a406d5c0 msgr2=0x7ff4a406fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff4a406d5c0 0x7ff4a406fa80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8100c20 tx=0x7ff4a8007400 comp rx=0 tx=0).stop 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 msgr2=0x7ff4b81021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b81021a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000bd70 tx=0x7ff4a000bda0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 shutdown_connections 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7ff4a406d5c0 0x7ff4a406fa80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4b80ffbc0 0x7ff4b8103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 --2- 172.21.15.39:0/2007681858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4b8100dc0 0x7ff4b81021a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 >> 172.21.15.39:0/2007681858 conn(0x7ff4b80fb160 msgr2=0x7ff4b8068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.297+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 shutdown_connections 2024-02-19T19:06:16.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.298+0000 7ff4bd2f4700 1 -- 172.21.15.39:0/2007681858 wait complete. 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: [19/Feb/2024:19:06:14] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: pgmap v3: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: [19/Feb/2024:19:06:14] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: [19/Feb/2024:19:06:14] ENGINE Bus STARTED 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:16.344 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:16.345 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/4006103647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:16.345 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:16 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/2312906055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:16.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 -- 172.21.15.39:0/2087543341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50101eb0 msgr2=0x7f4c50102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 --2- 172.21.15.39:0/2087543341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50101eb0 0x7f4c50102310 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009910 tx=0x7f4c44009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:16.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 -- 172.21.15.39:0/2087543341 shutdown_connections 2024-02-19T19:06:16.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 --2- 172.21.15.39:0/2087543341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50101eb0 0x7f4c50102310 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 --2- 172.21.15.39:0/2087543341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c50100cb0 0x7f4c501010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 -- 172.21.15.39:0/2087543341 >> 172.21.15.39:0/2087543341 conn(0x7f4c500fc230 msgr2=0x7f4c500fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:16.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 -- 172.21.15.39:0/2087543341 shutdown_connections 2024-02-19T19:06:16.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.403+0000 7f4c5552e700 1 -- 172.21.15.39:0/2087543341 wait complete. 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c5552e700 1 Processor -- start 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c5552e700 1 -- start start 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c5552e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 0x7f4c50074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c5552e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c50101eb0 0x7f4c50073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c5552e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c500736c0 con 0x7f4c50100cb0 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c5552e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50073800 con 0x7f4c50101eb0 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c4effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 0x7f4c50074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c4effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 0x7f4c50074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41360/0 (socket says 172.21.15.39:41360) 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.404+0000 7f4c4effd700 1 -- 172.21.15.39:0/3088771438 learned_addr learned my addr 172.21.15.39:0/3088771438 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c4e7fc700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c50101eb0 0x7f4c50073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c4effd700 1 -- 172.21.15.39:0/3088771438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c50101eb0 msgr2=0x7f4c50073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c4effd700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c50101eb0 0x7f4c50073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c4effd700 1 -- 172.21.15.39:0/3088771438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440095f0 con 0x7f4c50100cb0 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c4effd700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 0x7f4c50074b30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000d900 tx=0x7f4c4000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4000b920 con 0x7f4c50100cb0 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c50073ae0 con 0x7f4c50100cb0 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4c40010460 con 0x7f4c50100cb0 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4000f590 con 0x7f4c50100cb0 2024-02-19T19:06:16.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.405+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5019e550 con 0x7f4c50100cb0 2024-02-19T19:06:16.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.406+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c50109810 con 0x7f4c50100cb0 2024-02-19T19:06:16.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.406+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 35) v1 ==== 91245+0+0 (secure 0 0 0) 0x7f4c4000ba80 con 0x7f4c50100cb0 2024-02-19T19:06:16.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.407+0000 7f4c37fff700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4c3806d5c0 0x7f4c3806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:16.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.407+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4c4008cfd0 con 0x7f4c50100cb0 2024-02-19T19:06:16.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.407+0000 7f4c4e7fc700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4c3806d5c0 0x7f4c3806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:16.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.407+0000 7f4c4e7fc700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4c3806d5c0 0x7f4c3806fa80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400b5c0 tx=0x7f4c44005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:16.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.412+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c40056a30 con 0x7f4c50100cb0 2024-02-19T19:06:16.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.424+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 36) v1 ==== 91539+0+0 (secure 0 0 0) 0x7f4c4000bd30 con 0x7f4c50100cb0 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: [19/Feb/2024:19:06:14] ENGINE Serving on http://172.21.15.39:8765 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: pgmap v3: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: [19/Feb/2024:19:06:14] ENGINE Serving on https://172.21.15.39:7150 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: [19/Feb/2024:19:06:14] ENGINE Bus STARTED 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/4006103647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:16.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:16 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/2312906055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:16.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.704+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4c50109a00 con 0x7f4c50100cb0 2024-02-19T19:06:16.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.705+0000 7f4c37fff700 1 -- 172.21.15.39:0/3088771438 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f4c400875e0 con 0x7f4c50100cb0 2024-02-19T19:06:16.726 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:16.726 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:16.726 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4c3806d5c0 msgr2=0x7f4c3806fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4c3806d5c0 0x7f4c3806fa80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400b5c0 tx=0x7f4c44005ee0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 msgr2=0x7f4c50074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 0x7f4c50074b30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000d900 tx=0x7f4c4000dc10 comp rx=0 tx=0).stop 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 shutdown_connections 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4c3806d5c0 0x7f4c3806fa80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c50100cb0 0x7f4c50074b30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 --2- 172.21.15.39:0/3088771438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c50101eb0 0x7f4c50073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 >> 172.21.15.39:0/3088771438 conn(0x7f4c500fc230 msgr2=0x7f4c501050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 shutdown_connections 2024-02-19T19:06:16.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:16.707+0000 7f4c5552e700 1 -- 172.21.15.39:0/3088771438 wait complete. 2024-02-19T19:06:17.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:17 smithi039 ceph-mon[92224]: pgmap v4: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:17.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:17 smithi039 ceph-mon[92224]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:17.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:17 smithi039 ceph-mon[92224]: mgrmap e36: smithi039.ydmpuh(active, since 3s), standbys: smithi086.huubbu 2024-02-19T19:06:17.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:17 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3088771438' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:17.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:17.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:17.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:17 smithi086 ceph-mon[37629]: pgmap v4: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:17.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:17 smithi086 ceph-mon[37629]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:17.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:17 smithi086 ceph-mon[37629]: mgrmap e36: smithi039.ydmpuh(active, since 3s), standbys: smithi086.huubbu 2024-02-19T19:06:17.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:17 smithi086 ceph-mon[37629]: from='client.? 172.21.15.39:0/3088771438' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:17.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:17 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:17.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:17 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:19.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:18 smithi039 ceph-mon[92224]: pgmap v5: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:19.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:18 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:19.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:18 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:19.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:18 smithi039 ceph-mon[92224]: mgrmap e37: smithi039.ydmpuh(active, since 4s), standbys: smithi086.huubbu 2024-02-19T19:06:19.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:18 smithi086 ceph-mon[37629]: pgmap v5: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:19.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:18 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:19.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:18 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:19.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:18 smithi086 ceph-mon[37629]: mgrmap e37: smithi039.ydmpuh(active, since 4s), standbys: smithi086.huubbu 2024-02-19T19:06:20.574 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:20 smithi086 ceph-mon[37629]: pgmap v6: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:20.574 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:20 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:20.574 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:20 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:20.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:20 smithi039 ceph-mon[92224]: pgmap v6: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:06:20.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:20.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:06:21.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T19:06:21.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:21 smithi039 ceph-mon[92224]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T19:06:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:22.034 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:22.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:22.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:22.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:22.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:06:22.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.conf 2024-02-19T19:06:22.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:21 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.conf 2024-02-19T19:06:23.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: pgmap v7: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-19T19:06:23.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:06:23.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: pgmap v7: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 30 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: Updating smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: Updating smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:23.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:23.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:23.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:23 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: pgmap v8: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: Upgrade: Updating mon.smithi086 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:25.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 ceph-mon[37629]: Deploying daemon mon.smithi086 on smithi086 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: pgmap v8: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: Upgrade: Updating mon.smithi086 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:25.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:25 smithi039 ceph-mon[92224]: Deploying daemon mon.smithi086 on smithi086 2024-02-19T19:06:26.236 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:25 smithi086 systemd[1]: Stopping Ceph mon.smithi086 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:06:26.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:26 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086[37624]: 2024-02-19T19:06:26.215+0000 7fc540114700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi086 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:06:26.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:26 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086[37624]: 2024-02-19T19:06:26.215+0000 7fc540114700 -1 mon.smithi086@1(peon) e2 *** Got Signal Terminated *** 2024-02-19T19:06:26.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:26 smithi086 podman[82510]: 2024-02-19 19:06:26.246506937 +0000 UTC m=+0.156354898 container died 99602b1b0a0100daf9a823ae369fda615d7babcf7c6cdd30d36086cf4304f86c (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) 2024-02-19T19:06:26.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:26 smithi086 podman[82510]: 2024-02-19 19:06:26.633155284 +0000 UTC m=+0.543003241 container cleanup 99602b1b0a0100daf9a823ae369fda615d7babcf7c6cdd30d36086cf4304f86c (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086, RELEASE=HEAD, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1) 2024-02-19T19:06:26.879 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:26 smithi086 bash[82510]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086 2024-02-19T19:06:27.227 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:26 smithi086 podman[82524]: 2024-02-19 19:06:26.859157168 +0000 UTC m=+0.612407561 container remove 99602b1b0a0100daf9a823ae369fda615d7babcf7c6cdd30d36086cf4304f86c (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-02-19T19:06:27.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi086.service: Deactivated successfully. 2024-02-19T19:06:27.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 systemd[1]: Stopped Ceph mon.smithi086 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:06:27.228 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi086.service: Consumed 5.993s CPU time. 2024-02-19T19:06:27.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 systemd[1]: Starting Ceph mon.smithi086 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:06:27.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 podman[82655]: 2024-02-19 19:06:27.325831379 +0000 UTC m=+0.017609285 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:27.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 podman[82655]: 2024-02-19 19:06:27.451998172 +0000 UTC m=+0.143776060 container create 7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-19T19:06:28.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 podman[82655]: 2024-02-19 19:06:27.680205341 +0000 UTC m=+0.371983232 container init 7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 podman[82655]: 2024-02-19 19:06:27.683196682 +0000 UTC m=+0.374974571 container start 7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi086, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev), process ceph-mon, pid 2 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: pidfile_write: ignore empty --pid-file 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: load: jerasure load: lrc 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: RocksDB version: 7.9.2 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Git sha 0 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Compile date 2024-02-15 18:05:23 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: DB SUMMARY 2024-02-19T19:06:28.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: DB Session ID: MYQBTTRF6X2ESHTNGGKS 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: CURRENT file: CURRENT 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1403 Bytes 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi086/store.db dir, Total Num: 1, files: 000027.sst 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi086/store.db: 000025.log size: 4660806 ; 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.error_if_exists: 0 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.create_if_missing: 0 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.paranoid_checks: 1 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T19:06:28.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.env: 0x55955b13bae0 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.info_log: 0x55955ce8f760 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.statistics: (nil) 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.use_fsync: 0 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_log_file_size: 0 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.allow_fallocate: 1 2024-02-19T19:06:28.038 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.use_direct_reads: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.db_log_dir: 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.wal_dir: 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T19:06:28.039 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.write_buffer_manager: 0x55955ce63400 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.unordered_write: 0 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T19:06:28.040 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T19:06:28.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T19:06:28.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.row_cache: None 2024-02-19T19:06:28.041 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.wal_filter: None 2024-02-19T19:06:28.042 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T19:06:28.042 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.two_write_queues: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.wal_compression: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.atomic_flush: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.log_readahead_size: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T19:06:28.043 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_background_jobs: 2 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_background_compactions: -1 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_subcompactions: 1 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T19:06:28.044 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_open_files: -1 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_background_flushes: -1 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Compression algorithms supported: 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kZSTD supported: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kXpressCompression supported: 0 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kZlibCompression supported: 1 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kSnappyCompression supported: 1 2024-02-19T19:06:28.045 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kLZ4Compression supported: 1 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: kBZip2Compression supported: 0 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000010 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.merge_operator: 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_filter: None 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55955ce8f880) 2024-02-19T19:06:28.046 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_type: 0 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_index_type: 0 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_shortening: 1 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: checksum: 4 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: no_block_cache: 0 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache: 0x55955ce93090 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_options: 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: capacity : 536870912 2024-02-19T19:06:28.047 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: num_shard_bits : 4 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: strict_capacity_limit : 0 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_cache_compressed: (nil) 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: persistent_cache: (nil) 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size: 4096 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_size_deviation: 10 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_restart_interval: 16 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: index_block_restart_interval: 1 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: metadata_block_size: 4096 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: partition_filters: 0 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: use_delta_encoding: 1 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: filter_policy: bloomfilter 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: whole_key_filtering: 1 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: verify_compression: 0 2024-02-19T19:06:28.048 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: format_version: 5 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_index_compression: 1 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: block_align: 0 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_auto_readahead_size: 262144 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: prepopulate_block_cache: 0 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: initial_auto_readahead_size: 8192 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression: NoCompression 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T19:06:28.049 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.num_levels: 7 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:06:28.050 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T19:06:28.051 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T19:06:28.052 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.table_properties_collectors: 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.inplace_update_support: 0 2024-02-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.bloom_locality: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.max_successive_merges: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.ttl: 2592000 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T19:06:28.054 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.enable_blob_files: false 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.min_blob_size: 0 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12044, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c0f357e8-6f41-4436-b617-232f331df482 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369587712672, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-19T19:06:28.055 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369587722873, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2877272, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12049, "largest_seqno": 13179, "table_properties": {"data_size": 2871927, "index_size": 3110, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1221, "raw_key_size": 11377, "raw_average_key_size": 23, "raw_value_size": 2861592, "raw_average_value_size": 5986, "num_data_blocks": 142, "num_entries": 478, "num_filter_entries": 478, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708369587, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c0f357e8-6f41-4436-b617-232f331df482", "db_session_id": "MYQBTTRF6X2ESHTNGGKS", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369587722977, "job": 1, "event": "recovery_finished"} 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi086/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55955cf82000 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: DB pointer 0x55955cf6e000 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** DB Stats ** 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:06:28.056 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L0 1/0 2.74 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 673.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: L6 1/0 5.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Sum 2/0 8.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 673.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 673.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** Compaction Stats [default] ** 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:06:28.057 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 673.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Cumulative compaction: 0.00 GB write, 162.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Interval compaction: 0.00 GB write, 162.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T19:06:28.058 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Block cache BinnedLRUCache@0x55955ce93090#2 capacity: 512.00 MB usage: 4.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: starting mon.smithi086 rank 1 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi086 fsid 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???) e2 preinit fsid 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).mds e15 new map 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).mds e15 print_map 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: e15 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: legacy client fscid: 1 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Filesystem 'cephfs' (1) 2024-02-19T19:06:28.059 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: fs_name cephfs 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: epoch 15 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: tableserver 0 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: root 0 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: session_timeout 60 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: session_autoclose 300 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_file_size 1099511627776 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_xattr_size 65536 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: required_client_features {} 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: last_failure 0 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: last_failure_osd_epoch 46 2024-02-19T19:06:28.060 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: max_mds 1 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: in 0 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: up {0=24267} 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: failed 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: damaged 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: stopped 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: data_pools [3] 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: metadata_pool 2 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: inline_data enabled 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: balancer 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: bal_rank_mask -1 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: standby_count_wanted 1 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:28.061 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: Standby daemons: 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout: [mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 ceph-mon[82675]: mon.smithi086@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 bash[82655]: 7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085 2024-02-19T19:06:28.062 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:27 smithi086 systemd[1]: Started Ceph mon.smithi086 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:06:29.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:29 smithi039 ceph-mon[92224]: pgmap v9: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:06:29.784 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:29 smithi086 ceph-mon[82675]: pgmap v9: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:06:30.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:06:30.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: mon.smithi039 calling monitor election 2024-02-19T19:06:30.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: mon.smithi086 calling monitor election 2024-02-19T19:06:30.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: mon.smithi039 is new leader, mons smithi039,smithi086 in quorum (ranks 0,1) 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: monmap e3: 2 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: mgrmap e37: smithi039.ydmpuh(active, since 15s), standbys: smithi086.huubbu 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:30.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:30.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi039"}]: dispatch 2024-02-19T19:06:30.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: mon.smithi039 calling monitor election 2024-02-19T19:06:30.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: mon.smithi086 calling monitor election 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mon metadata", "id": "smithi086"}]: dispatch 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: mon.smithi039 is new leader, mons smithi039,smithi086 in quorum (ranks 0,1) 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: monmap e3: 2 mons at {smithi039=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],smithi086=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: mgrmap e37: smithi039.ydmpuh(active, since 15s), standbys: smithi086.huubbu 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:31.591 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:31 smithi086 ceph-mon[82675]: pgmap v11: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:06:31.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:31 smithi039 ceph-mon[92224]: pgmap v11: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:06:33.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:33 smithi039 ceph-mon[92224]: pgmap v12: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:06:33.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:33.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:33 smithi086 ceph-mon[82675]: pgmap v12: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 18 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:06:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:34.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:34 smithi039 ceph-mon[92224]: pgmap v13: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:34.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:34 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:34.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:34 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:34.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:34 smithi086 ceph-mon[82675]: pgmap v13: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:34.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:34 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:34.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:34 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:36.525 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:36 smithi086 ceph-mon[82675]: pgmap v14: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:36.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:36 smithi039 ceph-mon[92224]: pgmap v14: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:38.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:38.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:38.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:38.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:06:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: Reconfiguring mon.smithi039 (monmap changed)... 2024-02-19T19:06:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:37 smithi039 ceph-mon[92224]: Reconfiguring daemon mon.smithi039 on smithi039 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: Reconfiguring mon.smithi039 (monmap changed)... 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:38.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:37 smithi086 ceph-mon[82675]: Reconfiguring daemon mon.smithi039 on smithi039 2024-02-19T19:06:39.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:38 smithi039 ceph-mon[92224]: pgmap v15: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:39.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:38 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:39.182 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:38 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:39.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:38 smithi086 ceph-mon[82675]: pgmap v15: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:39.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:38 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:39.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:38 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:40.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:39 smithi039 ceph-mon[92224]: Reconfiguring mgr.smithi039.ydmpuh (monmap changed)... 2024-02-19T19:06:40.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:06:40.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:06:40.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:40.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:39 smithi039 ceph-mon[92224]: Reconfiguring daemon mgr.smithi039.ydmpuh on smithi039 2024-02-19T19:06:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:39 smithi086 ceph-mon[82675]: Reconfiguring mgr.smithi039.ydmpuh (monmap changed)... 2024-02-19T19:06:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi039.ydmpuh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:06:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:06:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:40.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:39 smithi086 ceph-mon[82675]: Reconfiguring daemon mgr.smithi039.ydmpuh on smithi039 2024-02-19T19:06:41.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: pgmap v16: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:41.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:41.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:41.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:41.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:41.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:06:41.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi039"}]: dispatch 2024-02-19T19:06:41.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:40 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: pgmap v16: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi039"}]: dispatch 2024-02-19T19:06:41.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:40 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:42.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:42 smithi086 ceph-mon[82675]: Reconfiguring ceph-exporter.smithi039 (monmap changed)... 2024-02-19T19:06:42.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:42 smithi086 ceph-mon[82675]: Unable to update caps for client.ceph-exporter.smithi039 2024-02-19T19:06:42.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:42 smithi086 ceph-mon[82675]: Reconfiguring daemon ceph-exporter.smithi039 on smithi039 2024-02-19T19:06:42.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:42 smithi039 ceph-mon[92224]: Reconfiguring ceph-exporter.smithi039 (monmap changed)... 2024-02-19T19:06:42.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:42 smithi039 ceph-mon[92224]: Unable to update caps for client.ceph-exporter.smithi039 2024-02-19T19:06:42.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:42 smithi039 ceph-mon[92224]: Reconfiguring daemon ceph-exporter.smithi039 on smithi039 2024-02-19T19:06:43.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:43 smithi086 ceph-mon[82675]: pgmap v17: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:43.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:43.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:43.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:06:43.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:43.380 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:43 smithi039 ceph-mon[92224]: pgmap v17: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:43.380 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:43.380 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:43.380 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:06:43.381 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:44.232 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:44 smithi039 ceph-mon[92224]: Reconfiguring crash.smithi039 (monmap changed)... 2024-02-19T19:06:44.232 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:44 smithi039 ceph-mon[92224]: Reconfiguring daemon crash.smithi039 on smithi039 2024-02-19T19:06:44.232 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:44.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:44 smithi086 ceph-mon[82675]: Reconfiguring crash.smithi039 (monmap changed)... 2024-02-19T19:06:44.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:44 smithi086 ceph-mon[82675]: Reconfiguring daemon crash.smithi039 on smithi039 2024-02-19T19:06:44.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:45 smithi039 ceph-mon[92224]: pgmap v18: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:45 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:45 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:45 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:06:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:45 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:45 smithi086 ceph-mon[82675]: pgmap v18: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:45 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:45 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:45 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:06:45.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:45 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:46.117 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:46 smithi039 ceph-mon[92224]: Reconfiguring osd.0 (monmap changed)... 2024-02-19T19:06:46.118 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:46 smithi039 ceph-mon[92224]: Reconfiguring daemon osd.0 on smithi039 2024-02-19T19:06:46.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:46 smithi086 ceph-mon[82675]: Reconfiguring osd.0 (monmap changed)... 2024-02-19T19:06:46.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:46 smithi086 ceph-mon[82675]: Reconfiguring daemon osd.0 on smithi039 2024-02-19T19:06:46.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 -- 172.21.15.39:0/4248558581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738101820 msgr2=0x7f1738103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:46.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 --2- 172.21.15.39:0/4248558581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738101820 0x7f1738103c10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1734009910 tx=0x7f1734009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:46.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 -- 172.21.15.39:0/4248558581 shutdown_connections 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 --2- 172.21.15.39:0/4248558581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738104150 0x7f1738106540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 --2- 172.21.15.39:0/4248558581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738101820 0x7f1738103c10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 -- 172.21.15.39:0/4248558581 >> 172.21.15.39:0/4248558581 conn(0x7f17380fb190 msgr2=0x7f17380fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 -- 172.21.15.39:0/4248558581 shutdown_connections 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 -- 172.21.15.39:0/4248558581 wait complete. 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 Processor -- start 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.818+0000 7f174017d700 1 -- start start 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f174017d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738101820 0x7f1738101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f174017d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 0x7f17380ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f174017d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380ffd10 con 0x7f1738104150 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f174017d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380ffe50 con 0x7f1738101820 2024-02-19T19:06:46.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 0x7f17380ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173df19700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738101820 0x7f1738101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 0x7f17380ff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52958/0 (socket says 172.21.15.39:52958) 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 -- 172.21.15.39:0/1545604769 learned_addr learned my addr 172.21.15.39:0/1545604769 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173df19700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738101820 0x7f1738101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:58346/0 (socket says 172.21.15.39:58346) 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 -- 172.21.15.39:0/1545604769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738101820 msgr2=0x7f1738101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738101820 0x7f1738101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 -- 172.21.15.39:0/1545604769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340095f0 con 0x7f1738104150 2024-02-19T19:06:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.819+0000 7f173d718700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 0x7f17380ff7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f172800c8a0 tx=0x7f172800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.820+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17280078e0 con 0x7f1738104150 2024-02-19T19:06:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.820+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1738100130 con 0x7f1738104150 2024-02-19T19:06:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.820+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f172800f450 con 0x7f1738104150 2024-02-19T19:06:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.820+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f172800e5e0 con 0x7f1738104150 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.820+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173819e4d0 con 0x7f1738104150 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.821+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1738190610 con 0x7f1738104150 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.822+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f1728007a40 con 0x7f1738104150 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.822+0000 7f172effd700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f172406d810 0x7f172406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.822+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f172808d470 con 0x7f1738104150 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.822+0000 7f173df19700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f172406d810 0x7f172406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.823+0000 7f173df19700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f172406d810 0x7f172406fcd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1734005100 tx=0x7f173401a040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:46.827+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f172805a2d0 con 0x7f1738104150 2024-02-19T19:06:47.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.056+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17380611d0 con 0x7f172406d810 2024-02-19T19:06:47.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.058+0000 7f172effd700 1 -- 172.21.15.39:0/1545604769 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f17380611d0 con 0x7f172406d810 2024-02-19T19:06:47.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f172406d810 msgr2=0x7f172406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f172406d810 0x7f172406fcd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1734005100 tx=0x7f173401a040 comp rx=0 tx=0).stop 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 msgr2=0x7f17380ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 0x7f17380ff7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f172800c8a0 tx=0x7f172800cbb0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 shutdown_connections 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f172406d810 0x7f172406fcd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1738101820 0x7f1738101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 --2- 172.21.15.39:0/1545604769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1738104150 0x7f17380ff7d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.061+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 >> 172.21.15.39:0/1545604769 conn(0x7f17380fb190 msgr2=0x7f17380fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.062+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 shutdown_connections 2024-02-19T19:06:47.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.062+0000 7f174017d700 1 -- 172.21.15.39:0/1545604769 wait complete. 2024-02-19T19:06:47.097 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 -- 172.21.15.39:0/3997212660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 msgr2=0x7f6d54101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 --2- 172.21.15.39:0/3997212660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54101220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009910 tx=0x7f6d3c009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 -- 172.21.15.39:0/3997212660 shutdown_connections 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 --2- 172.21.15.39:0/3997212660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54101220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 --2- 172.21.15.39:0/3997212660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d540ffbc0 0x7f6d540fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 -- 172.21.15.39:0/3997212660 >> 172.21.15.39:0/3997212660 conn(0x7f6d540fb160 msgr2=0x7f6d540fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 -- 172.21.15.39:0/3997212660 shutdown_connections 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.174+0000 7f6d59261700 1 -- 172.21.15.39:0/3997212660 wait complete. 2024-02-19T19:06:47.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d59261700 1 Processor -- start 2024-02-19T19:06:47.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d59261700 1 -- start start 2024-02-19T19:06:47.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d59261700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d540ffbc0 0x7f6d54195760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d59261700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d59261700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d541962c0 con 0x7f6d54100dc0 2024-02-19T19:06:47.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d59261700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d54196400 con 0x7f6d540ffbc0 2024-02-19T19:06:47.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d527fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d527fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54195ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52982/0 (socket says 172.21.15.39:52982) 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.175+0000 7f6d527fc700 1 -- 172.21.15.39:0/3236871979 learned_addr learned my addr 172.21.15.39:0/3236871979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d52ffd700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d540ffbc0 0x7f6d54195760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d527fc700 1 -- 172.21.15.39:0/3236871979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d540ffbc0 msgr2=0x7f6d54195760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d527fc700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d540ffbc0 0x7f6d54195760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d527fc700 1 -- 172.21.15.39:0/3236871979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c0095f0 con 0x7f6d54100dc0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d527fc700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54195ca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c006010 tx=0x7f6d3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d3c01d070 con 0x7f6d54100dc0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6d3c00f460 con 0x7f6d54100dc0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5419ae50 con 0x7f6d54100dc0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d3c021640 con 0x7f6d54100dc0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.176+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5419b310 con 0x7f6d54100dc0 2024-02-19T19:06:47.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.177+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5418f5d0 con 0x7f6d54100dc0 2024-02-19T19:06:47.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.178+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f6d3c00fab0 con 0x7f6d54100dc0 2024-02-19T19:06:47.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.178+0000 7f6d4bfff700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6d4006d9e0 0x7f6d4006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.178+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6d3c08ebb0 con 0x7f6d54100dc0 2024-02-19T19:06:47.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.178+0000 7f6d52ffd700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6d4006d9e0 0x7f6d4006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.178+0000 7f6d52ffd700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6d4006d9e0 0x7f6d4006fea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d54100c20 tx=0x7f6d440094f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:47.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.183+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d3c095080 con 0x7f6d54100dc0 2024-02-19T19:06:47.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.406+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d5402ceb0 con 0x7f6d4006d9e0 2024-02-19T19:06:47.427 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: pgmap v19: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:47.427 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:47.427 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:47.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: Reconfiguring osd.1 (monmap changed)... 2024-02-19T19:06:47.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:06:47.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:47.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:47 smithi039 ceph-mon[92224]: Reconfiguring daemon osd.1 on smithi039 2024-02-19T19:06:47.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.407+0000 7f6d4bfff700 1 -- 172.21.15.39:0/3236871979 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f6d5402ceb0 con 0x7f6d4006d9e0 2024-02-19T19:06:47.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6d4006d9e0 msgr2=0x7f6d4006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6d4006d9e0 0x7f6d4006fea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d54100c20 tx=0x7f6d440094f0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 msgr2=0x7f6d54195ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54195ca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c006010 tx=0x7f6d3c00be20 comp rx=0 tx=0).stop 2024-02-19T19:06:47.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 shutdown_connections 2024-02-19T19:06:47.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6d4006d9e0 0x7f6d4006fea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d540ffbc0 0x7f6d54195760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 --2- 172.21.15.39:0/3236871979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d54100dc0 0x7f6d54195ca0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 >> 172.21.15.39:0/3236871979 conn(0x7f6d540fb160 msgr2=0x7f6d541040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:47.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 shutdown_connections 2024-02-19T19:06:47.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.410+0000 7f6d59261700 1 -- 172.21.15.39:0/3236871979 wait complete. 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: pgmap v19: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: Reconfiguring osd.1 (monmap changed)... 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:47 smithi086 ceph-mon[82675]: Reconfiguring daemon osd.1 on smithi039 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- 172.21.15.39:0/502673028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 msgr2=0x7f49580fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 --2- 172.21.15.39:0/502673028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f49580fffd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4948009910 tx=0x7f4948009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- 172.21.15.39:0/502673028 shutdown_connections 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 --2- 172.21.15.39:0/502673028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 0x7f4958101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 --2- 172.21.15.39:0/502673028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f49580fffd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- 172.21.15.39:0/502673028 >> 172.21.15.39:0/502673028 conn(0x7f49580fb150 msgr2=0x7f49580fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- 172.21.15.39:0/502673028 shutdown_connections 2024-02-19T19:06:47.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- 172.21.15.39:0/502673028 wait complete. 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 Processor -- start 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- start start 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f4958103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 0x7f4958102190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581026d0 con 0x7f49580ffbb0 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.518+0000 7f495eac8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4958102810 con 0x7f4958100db0 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f4958103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f4958103b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53000/0 (socket says 172.21.15.39:53000) 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 -- 172.21.15.39:0/4276618564 learned_addr learned my addr 172.21.15.39:0/4276618564 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f4957fff700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 0x7f4958102190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 -- 172.21.15.39:0/4276618564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 msgr2=0x7f4958102190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 0x7f4958102190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 -- 172.21.15.39:0/4276618564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49480095f0 con 0x7f49580ffbb0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f4957fff700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 0x7f4958102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495c864700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f4958103b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f494800bda0 tx=0x7f494800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f494801d070 con 0x7f49580ffbb0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4948004c50 con 0x7f49580ffbb0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f494800f690 con 0x7f49580ffbb0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.519+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4958102a90 con 0x7f49580ffbb0 2024-02-19T19:06:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.520+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4958102f80 con 0x7f49580ffbb0 2024-02-19T19:06:47.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.521+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4958108790 con 0x7f49580ffbb0 2024-02-19T19:06:47.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.521+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f49480229b0 con 0x7f49580ffbb0 2024-02-19T19:06:47.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.522+0000 7f4955ffb700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f494006d9e0 0x7f494006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.522+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f494808eeb0 con 0x7f49580ffbb0 2024-02-19T19:06:47.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.522+0000 7f4957fff700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f494006d9e0 0x7f494006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.522+0000 7f4957fff700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f494006d9e0 0x7f494006fea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f494c0099d0 tx=0x7f494c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:47.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.526+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f494805c160 con 0x7f49580ffbb0 2024-02-19T19:06:47.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.741+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f495819ea80 con 0x7f494006d9e0 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.752+0000 7f4955ffb700 1 -- 172.21.15.39:0/4276618564 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f495819ea80 con 0x7f494006d9e0 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (8m) 28s ago 10m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (10m) 28s ago 10m 7763k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (9m) 15s ago 9m 7708k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (10m) 28s ago 10m 6660k - 18.2.1 94eadd990e0a aa5fee990892 2024-02-19T19:06:47.773 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (9m) 15s ago 9m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (8m) 28s ago 10m 86.2M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (4m) 28s ago 4m 25.5M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (4m) 28s ago 4m 14.2M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (4m) 15s ago 4m 29.1M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (4m) 15s ago 4m 16.4M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (91s) 28s ago 11m 471M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (57s) 15s ago 9m 448M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (43s) 28s ago 12m 28.9M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (20s) 15s ago 9m 21.7M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (10m) 28s ago 10m 14.2M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (9m) 15s ago 9m 15.8M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:06:47.774 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (8m) 28s ago 8m 52.4M 1216M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:06:47.775 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (7m) 28s ago 7m 51.1M 1216M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:06:47.775 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (7m) 28s ago 7m 49.5M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:06:47.775 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (7m) 15s ago 7m 49.4M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:06:47.775 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (6m) 15s ago 6m 47.0M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:06:47.775 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (6m) 15s ago 6m 49.0M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:06:47.775 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (63s) 28s ago 9m 47.2M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f494006d9e0 msgr2=0x7f494006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f494006d9e0 0x7f494006fea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f494c0099d0 tx=0x7f494c008040 comp rx=0 tx=0).stop 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 msgr2=0x7f4958103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f4958103b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f494800bda0 tx=0x7f494800be80 comp rx=0 tx=0).stop 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 shutdown_connections 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f494006d9e0 0x7f494006fea0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49580ffbb0 0x7f4958103b40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 --2- 172.21.15.39:0/4276618564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4958100db0 0x7f4958102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 >> 172.21.15.39:0/4276618564 conn(0x7f49580fb150 msgr2=0x7f4958068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 shutdown_connections 2024-02-19T19:06:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.754+0000 7f495eac8700 1 -- 172.21.15.39:0/4276618564 wait complete. 2024-02-19T19:06:47.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 -- 172.21.15.39:0/1020971441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3840ff9a0 msgr2=0x7fb3840ffdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 --2- 172.21.15.39:0/1020971441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3840ff9a0 0x7fb3840ffdc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb374009910 tx=0x7fb374009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:47.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 -- 172.21.15.39:0/1020971441 shutdown_connections 2024-02-19T19:06:47.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 --2- 172.21.15.39:0/1020971441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb384100ba0 0x7fb384101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 --2- 172.21.15.39:0/1020971441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb3840ff9a0 0x7fb3840ffdc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 -- 172.21.15.39:0/1020971441 >> 172.21.15.39:0/1020971441 conn(0x7fb3840faf40 msgr2=0x7fb3840fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:47.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 -- 172.21.15.39:0/1020971441 shutdown_connections 2024-02-19T19:06:47.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.860+0000 7fb388b9f700 1 -- 172.21.15.39:0/1020971441 wait complete. 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.861+0000 7fb388b9f700 1 Processor -- start 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.861+0000 7fb388b9f700 1 -- start start 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.861+0000 7fb388b9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3840ff9a0 0x7fb3840690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb388b9f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 0x7fb38419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb388b9f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384196e70 con 0x7fb384100ba0 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb388b9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38419d940 con 0x7fb3840ff9a0 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 0x7fb38419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 0x7fb38419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53020/0 (socket says 172.21.15.39:53020) 2024-02-19T19:06:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 -- 172.21.15.39:0/632761491 learned_addr learned my addr 172.21.15.39:0/632761491 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb3837fe700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3840ff9a0 0x7fb3840690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 -- 172.21.15.39:0/632761491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3840ff9a0 msgr2=0x7fb3840690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3840ff9a0 0x7fb3840690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 -- 172.21.15.39:0/632761491 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3740095f0 con 0x7fb384100ba0 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.862+0000 7fb382ffd700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 0x7fb38419d400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00e950 tx=0x7fb36c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.863+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c00cba0 con 0x7fb384100ba0 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.863+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb36c00eea0 con 0x7fb384100ba0 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.863+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38419dbf0 con 0x7fb384100ba0 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.863+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c0186e0 con 0x7fb384100ba0 2024-02-19T19:06:47.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.863+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38419e140 con 0x7fb384100ba0 2024-02-19T19:06:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.864+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb384108500 con 0x7fb384100ba0 2024-02-19T19:06:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.864+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7fb36c018910 con 0x7fb384100ba0 2024-02-19T19:06:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.865+0000 7fb380ff9700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb37006d9e0 0x7fb37006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.865+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb36c014070 con 0x7fb384100ba0 2024-02-19T19:06:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.865+0000 7fb3837fe700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb37006d9e0 0x7fb37006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.865+0000 7fb3837fe700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb37006d9e0 0x7fb37006fea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb37400b6f0 tx=0x7fb374005fb0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:47.869+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb36c052d20 con 0x7fb384100ba0 2024-02-19T19:06:48.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.163+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb38419e770 con 0x7fb384100ba0 2024-02-19T19:06:48.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.163+0000 7fb380ff9700 1 -- 172.21.15.39:0/632761491 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7fb36c05abe0 con 0x7fb384100ba0 2024-02-19T19:06:48.184 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:06:48.185 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-02-19T19:06:48.186 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:06:48.186 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:06:48.186 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:06:48.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb37006d9e0 msgr2=0x7fb37006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb37006d9e0 0x7fb37006fea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb37400b6f0 tx=0x7fb374005fb0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 msgr2=0x7fb38419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 0x7fb38419d400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00e950 tx=0x7fb36c00ec60 comp rx=0 tx=0).stop 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 shutdown_connections 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb37006d9e0 0x7fb37006fea0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3840ff9a0 0x7fb3840690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 --2- 172.21.15.39:0/632761491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb384100ba0 0x7fb38419d400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 >> 172.21.15.39:0/632761491 conn(0x7fb3840faf40 msgr2=0x7fb384103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.165+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 shutdown_connections 2024-02-19T19:06:48.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.166+0000 7fb388b9f700 1 -- 172.21.15.39:0/632761491 wait complete. 2024-02-19T19:06:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 -- 172.21.15.39:0/621122134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 msgr2=0x7f0e240fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/621122134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e240fffc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009910 tx=0x7f0e20009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 -- 172.21.15.39:0/621122134 shutdown_connections 2024-02-19T19:06:48.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/621122134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 0x7f0e24101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/621122134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e240fffc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 -- 172.21.15.39:0/621122134 >> 172.21.15.39:0/621122134 conn(0x7f0e240fb120 msgr2=0x7f0e240fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:48.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 -- 172.21.15.39:0/621122134 shutdown_connections 2024-02-19T19:06:48.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 -- 172.21.15.39:0/621122134 wait complete. 2024-02-19T19:06:48.299 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:48 smithi039 ceph-mon[92224]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:48.299 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:48.299 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:48.299 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:06:48.300 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:48.300 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:48 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/632761491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.277+0000 7f0e2c02a700 1 Processor -- start 2024-02-19T19:06:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e2c02a700 1 -- start start 2024-02-19T19:06:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e2c02a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e241953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e2c02a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 0x7f0e24195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e2c02a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e24195f30 con 0x7f0e240ffba0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e2c02a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e24196070 con 0x7f0e24100da0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e29dc6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e241953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e29dc6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e241953d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53046/0 (socket says 172.21.15.39:53046) 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e29dc6700 1 -- 172.21.15.39:0/3018309737 learned_addr learned my addr 172.21.15.39:0/3018309737 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e295c5700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 0x7f0e24195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e29dc6700 1 -- 172.21.15.39:0/3018309737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 msgr2=0x7f0e24195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e29dc6700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 0x7f0e24195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e29dc6700 1 -- 172.21.15.39:0/3018309737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e200095f0 con 0x7f0e240ffba0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.278+0000 7f0e295c5700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 0x7f0e24195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.279+0000 7f0e29dc6700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e241953d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000bda0 tx=0x7f0e2000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.279+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2001f070 con 0x7f0e240ffba0 2024-02-19T19:06:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.279+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0e20019610 con 0x7f0e240ffba0 2024-02-19T19:06:48.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.279+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2000f690 con 0x7f0e240ffba0 2024-02-19T19:06:48.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.279+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e24068e10 con 0x7f0e240ffba0 2024-02-19T19:06:48.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.279+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e24069300 con 0x7f0e240ffba0 2024-02-19T19:06:48.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.280+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e24108700 con 0x7f0e240ffba0 2024-02-19T19:06:48.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.281+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f0e20003a60 con 0x7f0e240ffba0 2024-02-19T19:06:48.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.281+0000 7f0e1affd700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e1006d9e0 0x7f0e1006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:48.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.281+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0e20090bb0 con 0x7f0e240ffba0 2024-02-19T19:06:48.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.281+0000 7f0e295c5700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e1006d9e0 0x7f0e1006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:48.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.282+0000 7f0e295c5700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e1006d9e0 0x7f0e1006fea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e14005950 tx=0x7f0e1400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.286+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e20097080 con 0x7f0e240ffba0 2024-02-19T19:06:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:48 smithi086 ceph-mon[82675]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:06:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:48 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/632761491' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:48.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.561+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0e2419aa00 con 0x7f0e240ffba0 2024-02-19T19:06:48.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.561+0000 7f0e1affd700 1 -- 172.21.15.39:0/3018309737 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f0e20028760 con 0x7f0e240ffba0 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:06:48.583 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:06:48.584 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:48.585 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:48.586 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:06:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:06:48.586 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:48.586 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e1006d9e0 msgr2=0x7f0e1006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e1006d9e0 0x7f0e1006fea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e14005950 tx=0x7f0e1400b410 comp rx=0 tx=0).stop 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 msgr2=0x7f0e241953d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e241953d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000bda0 tx=0x7f0e2000bdd0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 shutdown_connections 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e1006d9e0 0x7f0e1006fea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e240ffba0 0x7f0e241953d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.564+0000 7f0e2c02a700 1 --2- 172.21.15.39:0/3018309737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e24100da0 0x7f0e24195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.565+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 >> 172.21.15.39:0/3018309737 conn(0x7f0e240fb120 msgr2=0x7f0e24103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.565+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 shutdown_connections 2024-02-19T19:06:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.565+0000 7f0e2c02a700 1 -- 172.21.15.39:0/3018309737 wait complete. 2024-02-19T19:06:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:06:48.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 -- 172.21.15.39:0/358579507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2764100e10 msgr2=0x7f2764101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 --2- 172.21.15.39:0/358579507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2764100e10 0x7f2764101270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2758009910 tx=0x7f2758009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:48.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 -- 172.21.15.39:0/358579507 shutdown_connections 2024-02-19T19:06:48.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 --2- 172.21.15.39:0/358579507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2764100e10 0x7f2764101270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 --2- 172.21.15.39:0/358579507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27640ffc10 0x7f2764100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 -- 172.21.15.39:0/358579507 >> 172.21.15.39:0/358579507 conn(0x7f27640fb190 msgr2=0x7f27640fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:48.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 -- 172.21.15.39:0/358579507 shutdown_connections 2024-02-19T19:06:48.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 -- 172.21.15.39:0/358579507 wait complete. 2024-02-19T19:06:48.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.690+0000 7f2769d36700 1 Processor -- start 2024-02-19T19:06:48.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2769d36700 1 -- start start 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2769d36700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 0x7f2764196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2769d36700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764100e10 0x7f27641969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2769d36700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2764196fe0 con 0x7f27640ffc10 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2769d36700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2764197120 con 0x7f2764100e10 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 0x7f2764196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 0x7f2764196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53054/0 (socket says 172.21.15.39:53054) 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 -- 172.21.15.39:0/3452321842 learned_addr learned my addr 172.21.15.39:0/3452321842 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2762ffd700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764100e10 0x7f27641969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 -- 172.21.15.39:0/3452321842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764100e10 msgr2=0x7f27641969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764100e10 0x7f27641969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 -- 172.21.15.39:0/3452321842 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27580095f0 con 0x7f27640ffc10 2024-02-19T19:06:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f27637fe700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 0x7f2764196480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f275400d900 tx=0x7f275400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:48.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27540041f0 con 0x7f27640ffc10 2024-02-19T19:06:48.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276419bbd0 con 0x7f27640ffc10 2024-02-19T19:06:48.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2754004350 con 0x7f27640ffc10 2024-02-19T19:06:48.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.691+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2754003da0 con 0x7f27640ffc10 2024-02-19T19:06:48.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.692+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276419c120 con 0x7f27640ffc10 2024-02-19T19:06:48.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.692+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2764108700 con 0x7f27640ffc10 2024-02-19T19:06:48.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.693+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f2754009730 con 0x7f27640ffc10 2024-02-19T19:06:48.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.693+0000 7f2760ff9700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f274c06d9e0 0x7f274c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:48.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.693+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f275408d8b0 con 0x7f27640ffc10 2024-02-19T19:06:48.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.693+0000 7f2762ffd700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f274c06d9e0 0x7f274c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:48.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.694+0000 7f2762ffd700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f274c06d9e0 0x7f274c06fea0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2758005b40 tx=0x7f2758005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:48.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.698+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f275405a9c0 con 0x7f27640ffc10 2024-02-19T19:06:48.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.929+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2764068c90 con 0x7f274c06d9e0 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.930+0000 7f2760ff9700 1 -- 172.21.15.39:0/3452321842 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f2764068c90 con 0x7f274c06d9e0 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:06:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "mon" 2024-02-19T19:06:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:06:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "4/23 daemons upgraded", 2024-02-19T19:06:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading mon daemons", 2024-02-19T19:06:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:06:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f274c06d9e0 msgr2=0x7f274c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f274c06d9e0 0x7f274c06fea0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2758005b40 tx=0x7f2758005a90 comp rx=0 tx=0).stop 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 msgr2=0x7f2764196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 0x7f2764196480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f275400d900 tx=0x7f275400dcc0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 shutdown_connections 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f274c06d9e0 0x7f274c06fea0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27640ffc10 0x7f2764196480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 --2- 172.21.15.39:0/3452321842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2764100e10 0x7f27641969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.932+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 >> 172.21.15.39:0/3452321842 conn(0x7f27640fb190 msgr2=0x7f2764068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.933+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 shutdown_connections 2024-02-19T19:06:48.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:48.933+0000 7f2769d36700 1 -- 172.21.15.39:0/3452321842 wait complete. 2024-02-19T19:06:49.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 -- 172.21.15.39:0/3654932288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 msgr2=0x7f22101010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:49.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 --2- 172.21.15.39:0/3654932288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f22101010c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2200009910 tx=0x7f2200009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:49.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 -- 172.21.15.39:0/3654932288 shutdown_connections 2024-02-19T19:06:49.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 --2- 172.21.15.39:0/3654932288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2210101ea0 0x7f2210102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 --2- 172.21.15.39:0/3654932288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f22101010c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 -- 172.21.15.39:0/3654932288 >> 172.21.15.39:0/3654932288 conn(0x7f22100fc260 msgr2=0x7f22100fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.041+0000 7f221531e700 1 -- 172.21.15.39:0/3654932288 shutdown_connections 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 -- 172.21.15.39:0/3654932288 wait complete. 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 Processor -- start 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 -- start start 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f2210074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2210101ea0 0x7f2210073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22100736c0 con 0x7f2210100ca0 2024-02-19T19:06:49.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.042+0000 7f221531e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2210073800 con 0x7f2210101ea0 2024-02-19T19:06:49.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f2210074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f2210074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53074/0 (socket says 172.21.15.39:53074) 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 -- 172.21.15.39:0/1591693716 learned_addr learned my addr 172.21.15.39:0/1591693716 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220e7fc700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2210101ea0 0x7f2210073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 -- 172.21.15.39:0/1591693716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2210101ea0 msgr2=0x7f2210073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2210101ea0 0x7f2210073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 -- 172.21.15.39:0/1591693716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000095f0 con 0x7f2210100ca0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f220effd700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f2210074b30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2200004940 tx=0x7f2200004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.043+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220001d070 con 0x7f2210100ca0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.044+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2210073a80 con 0x7f2210100ca0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.044+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2200005dc0 con 0x7f2210100ca0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.044+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220000f690 con 0x7f2210100ca0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.044+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2210073f70 con 0x7f2210100ca0 2024-02-19T19:06:49.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.045+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2210109800 con 0x7f2210100ca0 2024-02-19T19:06:49.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.045+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f2200004bc0 con 0x7f2210100ca0 2024-02-19T19:06:49.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.045+0000 7f21f7fff700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f21f806d9e0 0x7f21f806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:49.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.046+0000 7f220e7fc700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f21f806d9e0 0x7f21f806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:49.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.046+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f220008ee40 con 0x7f2210100ca0 2024-02-19T19:06:49.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.046+0000 7f220e7fc700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f21f806d9e0 0x7f21f806fea0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2204005950 tx=0x7f220400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:49.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.050+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f220005c060 con 0x7f2210100ca0 2024-02-19T19:06:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.360+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f221019e9b0 con 0x7f2210100ca0 2024-02-19T19:06:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.360+0000 7f21f7fff700 1 -- 172.21.15.39:0/1591693716 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f2200027090 con 0x7f2210100ca0 2024-02-19T19:06:49.382 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:49.382 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:49.382 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f21f806d9e0 msgr2=0x7f21f806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f21f806d9e0 0x7f21f806fea0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2204005950 tx=0x7f220400b410 comp rx=0 tx=0).stop 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 msgr2=0x7f2210074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f2210074b30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2200004940 tx=0x7f2200004970 comp rx=0 tx=0).stop 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 shutdown_connections 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f21f806d9e0 0x7f21f806fea0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2210100ca0 0x7f2210074b30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 --2- 172.21.15.39:0/1591693716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2210101ea0 0x7f2210073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 >> 172.21.15.39:0/1591693716 conn(0x7f22100fc260 msgr2=0x7f22101050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:49.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 shutdown_connections 2024-02-19T19:06:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:06:49.363+0000 7f221531e700 1 -- 172.21.15.39:0/1591693716 wait complete. 2024-02-19T19:06:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:49 smithi039 ceph-mon[92224]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:49 smithi039 ceph-mon[92224]: pgmap v20: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:49 smithi039 ceph-mon[92224]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:49 smithi039 ceph-mon[92224]: Reconfiguring osd.2 (monmap changed)... 2024-02-19T19:06:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:49 smithi039 ceph-mon[92224]: Reconfiguring daemon osd.2 on smithi039 2024-02-19T19:06:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:49 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3018309737' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:49.534 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:49 smithi086 ceph-mon[82675]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:49 smithi086 ceph-mon[82675]: pgmap v20: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:49 smithi086 ceph-mon[82675]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:49 smithi086 ceph-mon[82675]: Reconfiguring osd.2 (monmap changed)... 2024-02-19T19:06:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:49 smithi086 ceph-mon[82675]: Reconfiguring daemon osd.2 on smithi039 2024-02-19T19:06:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:49 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3018309737' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:50.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:50 smithi086 ceph-mon[82675]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:50.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:50 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/1591693716' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:50.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:50.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:50.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:06:50.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:50.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:50 smithi039 ceph-mon[92224]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:50.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:50 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/1591693716' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:50.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:50.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:50.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:06:50.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:51.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:51 smithi039 ceph-mon[92224]: pgmap v21: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:51.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:51 smithi039 ceph-mon[92224]: Reconfiguring mds.cephfs.smithi039.omsmsr (monmap changed)... 2024-02-19T19:06:51.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:51 smithi039 ceph-mon[92224]: Reconfiguring daemon mds.cephfs.smithi039.omsmsr on smithi039 2024-02-19T19:06:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:51 smithi086 ceph-mon[82675]: pgmap v21: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:51 smithi086 ceph-mon[82675]: Reconfiguring mds.cephfs.smithi039.omsmsr (monmap changed)... 2024-02-19T19:06:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:51 smithi086 ceph-mon[82675]: Reconfiguring daemon mds.cephfs.smithi039.omsmsr on smithi039 2024-02-19T19:06:53.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: pgmap v22: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:53.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:53.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:53.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: Reconfiguring mds.cephfs.smithi039.aygwnv (monmap changed)... 2024-02-19T19:06:53.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:06:53.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:53.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:52 smithi039 ceph-mon[92224]: Reconfiguring daemon mds.cephfs.smithi039.aygwnv on smithi039 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: pgmap v22: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: Reconfiguring mds.cephfs.smithi039.aygwnv (monmap changed)... 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:53.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:52 smithi086 ceph-mon[82675]: Reconfiguring daemon mds.cephfs.smithi039.aygwnv on smithi039 2024-02-19T19:06:54.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: pgmap v23: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:54.718 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:54.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: pgmap v23: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: Reconfiguring ceph-exporter.smithi086 (monmap changed)... 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: Unable to update caps for client.ceph-exporter.smithi086 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi086"}]: dispatch 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:54 smithi039 ceph-mon[92224]: Reconfiguring daemon ceph-exporter.smithi086 on smithi086 2024-02-19T19:06:55.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:55.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: Reconfiguring ceph-exporter.smithi086 (monmap changed)... 2024-02-19T19:06:55.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:06:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi086", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:06:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: Unable to update caps for client.ceph-exporter.smithi086 2024-02-19T19:06:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi086"}]: dispatch 2024-02-19T19:06:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:54 smithi086 ceph-mon[82675]: Reconfiguring daemon ceph-exporter.smithi086 on smithi086 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: pgmap v24: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: Reconfiguring crash.smithi086 (monmap changed)... 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:56.618 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:56 smithi086 ceph-mon[82675]: Reconfiguring daemon crash.smithi086 on smithi086 2024-02-19T19:06:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: pgmap v24: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:06:56.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: Reconfiguring crash.smithi086 (monmap changed)... 2024-02-19T19:06:56.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:06:56.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:56.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:56 smithi039 ceph-mon[92224]: Reconfiguring daemon crash.smithi086 on smithi086 2024-02-19T19:06:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: Reconfiguring mgr.smithi086.huubbu (monmap changed)... 2024-02-19T19:06:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:06:58.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:06:58.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:58.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:58 smithi086 ceph-mon[82675]: Reconfiguring daemon mgr.smithi086.huubbu on smithi086 2024-02-19T19:06:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:58.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: Reconfiguring mgr.smithi086.huubbu (monmap changed)... 2024-02-19T19:06:58.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi086.huubbu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:06:58.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:06:58.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:58.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:58 smithi039 ceph-mon[92224]: Reconfiguring daemon mgr.smithi086.huubbu on smithi086 2024-02-19T19:06:59.389 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: pgmap v25: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:59.389 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:59.389 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:59.389 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:59.389 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:59.389 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:59.390 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:06:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: pgmap v25: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:06:59.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:06:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:00.415 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:00 smithi086 ceph-mon[82675]: Reconfiguring mon.smithi086 (monmap changed)... 2024-02-19T19:07:00.415 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:00 smithi086 ceph-mon[82675]: Reconfiguring daemon mon.smithi086 on smithi086 2024-02-19T19:07:00.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:00 smithi039 ceph-mon[92224]: Reconfiguring mon.smithi086 (monmap changed)... 2024-02-19T19:07:00.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:00 smithi039 ceph-mon[92224]: Reconfiguring daemon mon.smithi086 on smithi086 2024-02-19T19:07:01.315 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:01 smithi086 ceph-mon[82675]: pgmap v26: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:01.315 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:01 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:01.315 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:01 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:01.315 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:01 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:07:01.315 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:01 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:01.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:01 smithi039 ceph-mon[92224]: pgmap v26: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:01.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:01 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:01.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:01 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:01.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:01 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:07:01.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:01 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:02.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:02 smithi039 ceph-mon[92224]: Reconfiguring osd.3 (monmap changed)... 2024-02-19T19:07:02.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:02 smithi039 ceph-mon[92224]: Reconfiguring daemon osd.3 on smithi086 2024-02-19T19:07:02.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:02 smithi086 ceph-mon[82675]: Reconfiguring osd.3 (monmap changed)... 2024-02-19T19:07:02.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:02 smithi086 ceph-mon[82675]: Reconfiguring daemon osd.3 on smithi086 2024-02-19T19:07:03.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: pgmap v27: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:03.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:03.434 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:03.435 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: Reconfiguring osd.4 (monmap changed)... 2024-02-19T19:07:03.435 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:07:03.435 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:03.435 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:03 smithi086 ceph-mon[82675]: Reconfiguring daemon osd.4 on smithi086 2024-02-19T19:07:03.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: pgmap v27: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:03.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:03.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:03.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: Reconfiguring osd.4 (monmap changed)... 2024-02-19T19:07:03.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:07:03.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:03.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:03 smithi039 ceph-mon[92224]: Reconfiguring daemon osd.4 on smithi086 2024-02-19T19:07:05.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: pgmap v28: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:05.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:05.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:05.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: Reconfiguring osd.5 (monmap changed)... 2024-02-19T19:07:05.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:07:05.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:05.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:04 smithi086 ceph-mon[82675]: Reconfiguring daemon osd.5 on smithi086 2024-02-19T19:07:05.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: pgmap v28: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:05.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:05.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:05.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: Reconfiguring osd.5 (monmap changed)... 2024-02-19T19:07:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:07:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:05.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:04 smithi039 ceph-mon[92224]: Reconfiguring daemon osd.5 on smithi086 2024-02-19T19:07:06.873 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: pgmap v29: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:06.873 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:06.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:06.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: Reconfiguring mds.cephfs.smithi086.euvbgw (monmap changed)... 2024-02-19T19:07:06.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:07:06.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:06.874 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:06 smithi086 ceph-mon[82675]: Reconfiguring daemon mds.cephfs.smithi086.euvbgw on smithi086 2024-02-19T19:07:06.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: pgmap v29: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:06.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:06.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:06.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: Reconfiguring mds.cephfs.smithi086.euvbgw (monmap changed)... 2024-02-19T19:07:06.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:07:06.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:06.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:06 smithi039 ceph-mon[92224]: Reconfiguring daemon mds.cephfs.smithi086.euvbgw on smithi086 2024-02-19T19:07:08.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:08 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:08.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:08 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:08.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:08 smithi039 ceph-mon[92224]: Reconfiguring mds.cephfs.smithi086.idtugy (monmap changed)... 2024-02-19T19:07:08.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:08 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:07:08.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:08 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:08.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:08 smithi039 ceph-mon[92224]: Reconfiguring daemon mds.cephfs.smithi086.idtugy on smithi086 2024-02-19T19:07:08.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:08 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:08.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:08 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:08.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:08 smithi086 ceph-mon[82675]: Reconfiguring mds.cephfs.smithi086.idtugy (monmap changed)... 2024-02-19T19:07:08.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:08 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:07:08.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:08 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:08.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:08 smithi086 ceph-mon[82675]: Reconfiguring daemon mds.cephfs.smithi086.idtugy on smithi086 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: pgmap v30: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi039"}]: dispatch 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi039"}]': finished 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]: dispatch 2024-02-19T19:07:09.609 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]': finished 2024-02-19T19:07:09.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: pgmap v30: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:09.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:09.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:09.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:09.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:09.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:09.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:09.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi039"}]: dispatch 2024-02-19T19:07:09.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi039"}]': finished 2024-02-19T19:07:09.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]: dispatch 2024-02-19T19:07:09.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi086"}]': finished 2024-02-19T19:07:10.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:10 smithi086 ceph-mon[82675]: Upgrade: Setting container_image for all mon 2024-02-19T19:07:10.642 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:10 smithi039 ceph-mon[92224]: Upgrade: Setting container_image for all mon 2024-02-19T19:07:11.616 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:11 smithi039 ceph-mon[92224]: pgmap v31: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:11.616 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:11 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:11.616 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:11 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:07:11.616 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:11 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:11.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:11 smithi086 ceph-mon[82675]: pgmap v31: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:11.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:11 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:11.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:11 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi039", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:07:11.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:11 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:12.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:12 smithi039 ceph-mon[92224]: Upgrade: Updating crash.smithi039 (1/2) 2024-02-19T19:07:12.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:12 smithi039 ceph-mon[92224]: Deploying daemon crash.smithi039 on smithi039 2024-02-19T19:07:12.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:12 smithi086 ceph-mon[82675]: Upgrade: Updating crash.smithi039 (1/2) 2024-02-19T19:07:12.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:12 smithi086 ceph-mon[82675]: Deploying daemon crash.smithi039 on smithi039 2024-02-19T19:07:13.694 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:13 smithi039 ceph-mon[92224]: pgmap v32: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:13.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:13 smithi086 ceph-mon[82675]: pgmap v32: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:14.492 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:14.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:15.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:15 smithi086 ceph-mon[82675]: pgmap v33: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:15.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:15 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:15.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:15 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:15.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:15 smithi039 ceph-mon[92224]: pgmap v33: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:15.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:15 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:15.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:15 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:17.640 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:17 smithi086 ceph-mon[82675]: pgmap v34: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:17.640 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:17 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:17.640 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:17 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:07:17.640 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:17 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:17.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:17 smithi039 ceph-mon[92224]: pgmap v34: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:17.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:17.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi086", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:07:17.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:18.699 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:18 smithi086 ceph-mon[82675]: Upgrade: Updating crash.smithi086 (2/2) 2024-02-19T19:07:18.699 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:18 smithi086 ceph-mon[82675]: Deploying daemon crash.smithi086 on smithi086 2024-02-19T19:07:18.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:18 smithi039 ceph-mon[92224]: Upgrade: Updating crash.smithi086 (2/2) 2024-02-19T19:07:18.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:18 smithi039 ceph-mon[92224]: Deploying daemon crash.smithi086 on smithi086 2024-02-19T19:07:19.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 -- 172.21.15.39:0/889944633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51501017f0 msgr2=0x7f5150103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:19.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 --2- 172.21.15.39:0/889944633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51501017f0 0x7f5150103be0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f514800b0a0 tx=0x7f514800b3b0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 -- 172.21.15.39:0/889944633 shutdown_connections 2024-02-19T19:07:19.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 --2- 172.21.15.39:0/889944633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5150104120 0x7f5150106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 --2- 172.21.15.39:0/889944633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51501017f0 0x7f5150103be0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 -- 172.21.15.39:0/889944633 >> 172.21.15.39:0/889944633 conn(0x7f51500fb160 msgr2=0x7f51500fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 -- 172.21.15.39:0/889944633 shutdown_connections 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.490+0000 7f5155899700 1 -- 172.21.15.39:0/889944633 wait complete. 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.491+0000 7f5155899700 1 Processor -- start 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.491+0000 7f5155899700 1 -- start start 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.491+0000 7f5155899700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51501017f0 0x7f5150101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.491+0000 7f5155899700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 0x7f51500ff7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.491+0000 7f5155899700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51500ffce0 con 0x7f5150104120 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.491+0000 7f5155899700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51500ffe20 con 0x7f51501017f0 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 0x7f51500ff7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 0x7f51500ff7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58210/0 (socket says 172.21.15.39:58210) 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 -- 172.21.15.39:0/1952837070 learned_addr learned my addr 172.21.15.39:0/1952837070 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514effd700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51501017f0 0x7f5150101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 -- 172.21.15.39:0/1952837070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51501017f0 msgr2=0x7f5150101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51501017f0 0x7f5150101150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 -- 172.21.15.39:0/1952837070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5148009d00 con 0x7f5150104120 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f514e7fc700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 0x7f51500ff7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f513800e920 tx=0x7f513800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f513800c9d0 con 0x7f5150104120 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5138010430 con 0x7f5150104120 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5138018650 con 0x7f5150104120 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5150100100 con 0x7f5150104120 2024-02-19T19:07:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.492+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5150100600 con 0x7f5150104120 2024-02-19T19:07:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.493+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5150190640 con 0x7f5150104120 2024-02-19T19:07:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.494+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f51380105a0 con 0x7f5150104120 2024-02-19T19:07:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.494+0000 7f5154897700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f513c06d9e0 0x7f513c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.495+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5138014070 con 0x7f5150104120 2024-02-19T19:07:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.495+0000 7f514effd700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f513c06d9e0 0x7f513c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.495+0000 7f514effd700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f513c06d9e0 0x7f513c06fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5148009fd0 tx=0x7f51480060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.500+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5138052d20 con 0x7f5150104120 2024-02-19T19:07:19.717 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:19 smithi086 ceph-mon[82675]: pgmap v35: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:19.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:19 smithi039 ceph-mon[92224]: pgmap v35: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:19.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.730+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f515002cfe0 con 0x7f513c06d9e0 2024-02-19T19:07:19.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.732+0000 7f5154897700 1 -- 172.21.15.39:0/1952837070 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f515002cfe0 con 0x7f513c06d9e0 2024-02-19T19:07:19.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f513c06d9e0 msgr2=0x7f513c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:19.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f513c06d9e0 0x7f513c06fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5148009fd0 tx=0x7f51480060c0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 msgr2=0x7f51500ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:19.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 0x7f51500ff7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f513800e920 tx=0x7f513800ec30 comp rx=0 tx=0).stop 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 shutdown_connections 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f513c06d9e0 0x7f513c06fea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51501017f0 0x7f5150101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 --2- 172.21.15.39:0/1952837070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5150104120 0x7f51500ff7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 >> 172.21.15.39:0/1952837070 conn(0x7f51500fb160 msgr2=0x7f51500fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 shutdown_connections 2024-02-19T19:07:19.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.734+0000 7f5155899700 1 -- 172.21.15.39:0/1952837070 wait complete. 2024-02-19T19:07:19.764 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 -- 172.21.15.39:0/1210183383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 msgr2=0x7f5b84101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 --2- 172.21.15.39:0/1210183383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b84101270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009910 tx=0x7f5b78009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 -- 172.21.15.39:0/1210183383 shutdown_connections 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 --2- 172.21.15.39:0/1210183383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b84101270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 --2- 172.21.15.39:0/1210183383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 0x7f5b84100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 -- 172.21.15.39:0/1210183383 >> 172.21.15.39:0/1210183383 conn(0x7f5b840fb190 msgr2=0x7f5b840fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 -- 172.21.15.39:0/1210183383 shutdown_connections 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.842+0000 7f5b88c68700 1 -- 172.21.15.39:0/1210183383 wait complete. 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b88c68700 1 Processor -- start 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b88c68700 1 -- start start 2024-02-19T19:07:19.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b88c68700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 0x7f5b84103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b88c68700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b841021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b88c68700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84102730 con 0x7f5b84100e10 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b88c68700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84102870 con 0x7f5b840ffc10 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b81d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b841021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b8259c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 0x7f5b84103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b81d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b841021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58236/0 (socket says 172.21.15.39:58236) 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.843+0000 7f5b81d9b700 1 -- 172.21.15.39:0/3206533174 learned_addr learned my addr 172.21.15.39:0/3206533174 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b8259c700 1 -- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 msgr2=0x7f5b841021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b8259c700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b841021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b8259c700 1 -- 172.21.15.39:0/3206533174 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b780095f0 con 0x7f5b840ffc10 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b81d9b700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b841021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b8259c700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 0x7f5b84103ba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400e940 tx=0x7f5b7400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b74004200 con 0x7f5b840ffc10 2024-02-19T19:07:19.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b84102b50 con 0x7f5b840ffc10 2024-02-19T19:07:19.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b74004360 con 0x7f5b840ffc10 2024-02-19T19:07:19.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8419e530 con 0x7f5b840ffc10 2024-02-19T19:07:19.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.844+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b74003e40 con 0x7f5b840ffc10 2024-02-19T19:07:19.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.845+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64005320 con 0x7f5b840ffc10 2024-02-19T19:07:19.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.846+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f5b7400c7c0 con 0x7f5b840ffc10 2024-02-19T19:07:19.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.846+0000 7f5b737fe700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5b6c06d960 0x7f5b6c06fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:19.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.846+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5b74014070 con 0x7f5b840ffc10 2024-02-19T19:07:19.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.846+0000 7f5b81d9b700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5b6c06d960 0x7f5b6c06fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:19.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.846+0000 7f5b81d9b700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5b6c06d960 0x7f5b6c06fe20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009fd0 tx=0x7f5b78005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:19.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:19.855+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b7405ac00 con 0x7f5b840ffc10 2024-02-19T19:07:20.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.079+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b64000bf0 con 0x7f5b6c06d960 2024-02-19T19:07:20.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.080+0000 7f5b737fe700 1 -- 172.21.15.39:0/3206533174 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f5b64000bf0 con 0x7f5b6c06d960 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5b6c06d960 msgr2=0x7f5b6c06fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5b6c06d960 0x7f5b6c06fe20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009fd0 tx=0x7f5b78005a90 comp rx=0 tx=0).stop 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 msgr2=0x7f5b84103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 0x7f5b84103ba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400e940 tx=0x7f5b7400ec50 comp rx=0 tx=0).stop 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 shutdown_connections 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5b6c06d960 0x7f5b6c06fe20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b840ffc10 0x7f5b84103ba0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 --2- 172.21.15.39:0/3206533174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b84100e10 0x7f5b841021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 >> 172.21.15.39:0/3206533174 conn(0x7f5b840fb190 msgr2=0x7f5b84068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 shutdown_connections 2024-02-19T19:07:20.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.083+0000 7f5b88c68700 1 -- 172.21.15.39:0/3206533174 wait complete. 2024-02-19T19:07:20.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 -- 172.21.15.39:0/1402903194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 msgr2=0x7f7488100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 --2- 172.21.15.39:0/1402903194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488100030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7478009910 tx=0x7f7478009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:20.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 -- 172.21.15.39:0/1402903194 shutdown_connections 2024-02-19T19:07:20.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 --2- 172.21.15.39:0/1402903194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7488100e10 0x7f7488101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 --2- 172.21.15.39:0/1402903194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488100030 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 -- 172.21.15.39:0/1402903194 >> 172.21.15.39:0/1402903194 conn(0x7f74880fb190 msgr2=0x7f74880fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 -- 172.21.15.39:0/1402903194 shutdown_connections 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.191+0000 7f748e7a0700 1 -- 172.21.15.39:0/1402903194 wait complete. 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f748e7a0700 1 Processor -- start 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f748e7a0700 1 -- start start 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f748e7a0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f748e7a0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7488100e10 0x7f74881021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f748e7a0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488102730 con 0x7f74880ffc10 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f748e7a0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7488102870 con 0x7f7488100e10 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f7487fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f7487fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58252/0 (socket says 172.21.15.39:58252) 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f7487fff700 1 -- 172.21.15.39:0/3005619321 learned_addr learned my addr 172.21.15.39:0/3005619321 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f74877fe700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7488100e10 0x7f74881021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f7487fff700 1 -- 172.21.15.39:0/3005619321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7488100e10 msgr2=0x7f74881021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f7487fff700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7488100e10 0x7f74881021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.192+0000 7f7487fff700 1 -- 172.21.15.39:0/3005619321 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74780095f0 con 0x7f74880ffc10 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.193+0000 7f7487fff700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488103ba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7478000c00 tx=0x7f747800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.193+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f747801d070 con 0x7f74880ffc10 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.193+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7478005dc0 con 0x7f74880ffc10 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.193+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7478022470 con 0x7f74880ffc10 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.193+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7488102af0 con 0x7f74880ffc10 2024-02-19T19:07:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.193+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7488102fe0 con 0x7f74880ffc10 2024-02-19T19:07:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.194+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7488108700 con 0x7f74880ffc10 2024-02-19T19:07:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.194+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f7478003810 con 0x7f74880ffc10 2024-02-19T19:07:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.195+0000 7f74857fa700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f747006d9e0 0x7f747006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.195+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f747808eb50 con 0x7f74880ffc10 2024-02-19T19:07:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.195+0000 7f74877fe700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f747006d9e0 0x7f747006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.195+0000 7f74877fe700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f747006d9e0 0x7f747006fea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f747c009c60 tx=0x7f747c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:20.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.199+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f747805bce0 con 0x7f74880ffc10 2024-02-19T19:07:20.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.412+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7488068c90 con 0x7f747006d9e0 2024-02-19T19:07:20.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.423+0000 7f74857fa700 1 -- 172.21.15.39:0/3005619321 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7488068c90 con 0x7f747006d9e0 2024-02-19T19:07:20.443 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:07:20.443 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (9m) 60s ago 11m 25.2M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (11m) 60s ago 11m 7763k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (10m) 48s ago 10m 7708k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 starting - - - - 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (10m) 48s ago 9m 6665k - 18.2.1 94eadd990e0a 88f72d592626 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (9m) 60s ago 10m 86.2M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (5m) 60s ago 5m 25.5M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (5m) 60s ago 5m 14.2M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (5m) 48s ago 5m 29.1M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (5m) 48s ago 5m 16.4M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (2m) 60s ago 12m 471M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (90s) 48s ago 9m 448M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (76s) 60s ago 12m 28.9M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (53s) 48s ago 9m 21.7M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:07:20.444 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (11m) 60s ago 11m 14.2M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (9m) 48s ago 9m 15.8M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (9m) 60s ago 9m 52.4M 1216M 18.2.1 94eadd990e0a a841adabd463 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (8m) 60s ago 8m 51.1M 1216M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (8m) 60s ago 8m 49.5M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (7m) 48s ago 7m 49.4M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (7m) 48s ago 7m 47.0M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (6m) 48s ago 6m 49.0M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:07:20.445 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (96s) 60s ago 10m 47.2M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f747006d9e0 msgr2=0x7f747006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f747006d9e0 0x7f747006fea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f747c009c60 tx=0x7f747c009260 comp rx=0 tx=0).stop 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 msgr2=0x7f7488103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488103ba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7478000c00 tx=0x7f747800f690 comp rx=0 tx=0).stop 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 shutdown_connections 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f747006d9e0 0x7f747006fea0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74880ffc10 0x7f7488103ba0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 --2- 172.21.15.39:0/3005619321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7488100e10 0x7f74881021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 >> 172.21.15.39:0/3005619321 conn(0x7f74880fb190 msgr2=0x7f7488068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 shutdown_connections 2024-02-19T19:07:20.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.425+0000 7f748e7a0700 1 -- 172.21.15.39:0/3005619321 wait complete. 2024-02-19T19:07:20.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 -- 172.21.15.39:0/2152251809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d780ffbc0 msgr2=0x7f4d780fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/2152251809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d780ffbc0 0x7f4d780fffe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009910 tx=0x7f4d6c009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:20.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 -- 172.21.15.39:0/2152251809 shutdown_connections 2024-02-19T19:07:20.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/2152251809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d78100dc0 0x7f4d78101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/2152251809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d780ffbc0 0x7f4d780fffe0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 -- 172.21.15.39:0/2152251809 >> 172.21.15.39:0/2152251809 conn(0x7f4d780fb160 msgr2=0x7f4d780fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:20.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 -- 172.21.15.39:0/2152251809 shutdown_connections 2024-02-19T19:07:20.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.529+0000 7f4d7daf7700 1 -- 172.21.15.39:0/2152251809 wait complete. 2024-02-19T19:07:20.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d7daf7700 1 Processor -- start 2024-02-19T19:07:20.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d7daf7700 1 -- start start 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d7daf7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d780ffbc0 0x7f4d78103b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d7daf7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 0x7f4d781021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d7daf7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d781026e0 con 0x7f4d78100dc0 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d7daf7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d78102820 con 0x7f4d780ffbc0 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d76ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 0x7f4d781021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d76ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 0x7f4d781021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58262/0 (socket says 172.21.15.39:58262) 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d76ffd700 1 -- 172.21.15.39:0/681766254 learned_addr learned my addr 172.21.15.39:0/681766254 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.530+0000 7f4d777fe700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d780ffbc0 0x7f4d78103b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d76ffd700 1 -- 172.21.15.39:0/681766254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d780ffbc0 msgr2=0x7f4d78103b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d76ffd700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d780ffbc0 0x7f4d78103b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d76ffd700 1 -- 172.21.15.39:0/681766254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c0095f0 con 0x7f4d78100dc0 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d76ffd700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 0x7f4d781021a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800b4d0 tx=0x7f4d6800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:20.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d68011800 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4d68011e10 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d6800f2e0 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d78102b00 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.531+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7819e8f0 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.532+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d78068af0 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.533+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f4d68010380 con 0x7f4d78100dc0 2024-02-19T19:07:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.533+0000 7f4d74ff9700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4d6006d9d0 0x7f4d6006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.533+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4d6808d200 con 0x7f4d78100dc0 2024-02-19T19:07:20.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.533+0000 7f4d777fe700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4d6006d9d0 0x7f4d6006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.534+0000 7f4d777fe700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4d6006d9d0 0x7f4d6006fe90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009fd0 tx=0x7f4d6c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:20.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.538+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d6805a420 con 0x7f4d78100dc0 2024-02-19T19:07:20.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.826+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4d7819ea80 con 0x7f4d78100dc0 2024-02-19T19:07:20.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.827+0000 7f4d74ff9700 1 -- 172.21.15.39:0/681766254 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f4d68059fb0 con 0x7f4d78100dc0 2024-02-19T19:07:20.848 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:07:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:07:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:07:20.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:07:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:07:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:07:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:07:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:07:20.850 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:07:20.851 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:07:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4d6006d9d0 msgr2=0x7f4d6006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4d6006d9d0 0x7f4d6006fe90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009fd0 tx=0x7f4d6c00b540 comp rx=0 tx=0).stop 2024-02-19T19:07:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 msgr2=0x7f4d781021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 0x7f4d781021a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800b4d0 tx=0x7f4d6800b7e0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 shutdown_connections 2024-02-19T19:07:20.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4d6006d9d0 0x7f4d6006fe90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d780ffbc0 0x7f4d78103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 --2- 172.21.15.39:0/681766254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d78100dc0 0x7f4d781021a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.829+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 >> 172.21.15.39:0/681766254 conn(0x7f4d780fb160 msgr2=0x7f4d78068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:20.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.830+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 shutdown_connections 2024-02-19T19:07:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.830+0000 7f4d7daf7700 1 -- 172.21.15.39:0/681766254 wait complete. 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 -- 172.21.15.39:0/1538138800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 msgr2=0x7f80a40fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 --2- 172.21.15.39:0/1538138800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40fffe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8094009910 tx=0x7f8094009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 -- 172.21.15.39:0/1538138800 shutdown_connections 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 --2- 172.21.15.39:0/1538138800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a4100dc0 0x7f80a4101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 --2- 172.21.15.39:0/1538138800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40fffe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 -- 172.21.15.39:0/1538138800 >> 172.21.15.39:0/1538138800 conn(0x7f80a40fb160 msgr2=0x7f80a40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 -- 172.21.15.39:0/1538138800 shutdown_connections 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.939+0000 7f80a9a23700 1 -- 172.21.15.39:0/1538138800 wait complete. 2024-02-19T19:07:20.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a9a23700 1 Processor -- start 2024-02-19T19:07:20.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a9a23700 1 -- start start 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a9a23700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a9a23700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a4100dc0 0x7f80a419b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a9a23700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a419bb00 con 0x7f80a40ffbc0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a9a23700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a419bc40 con 0x7f80a4100dc0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58278/0 (socket says 172.21.15.39:58278) 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a2ffd700 1 -- 172.21.15.39:0/4090519763 learned_addr learned my addr 172.21.15.39:0/4090519763 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a27fc700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a4100dc0 0x7f80a419b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a2ffd700 1 -- 172.21.15.39:0/4090519763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a4100dc0 msgr2=0x7f80a419b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a2ffd700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a4100dc0 0x7f80a419b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.940+0000 7f80a2ffd700 1 -- 172.21.15.39:0/4090519763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940095f0 con 0x7f80a40ffbc0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.941+0000 7f80a2ffd700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40699d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f809400bda0 tx=0x7f809400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.941+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f809401d070 con 0x7f80a40ffbc0 2024-02-19T19:07:20.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.941+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f809400f460 con 0x7f80a40ffbc0 2024-02-19T19:07:20.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.941+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a419bec0 con 0x7f80a40ffbc0 2024-02-19T19:07:20.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.941+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8094021640 con 0x7f80a40ffbc0 2024-02-19T19:07:20.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.941+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a419c330 con 0x7f80a40ffbc0 2024-02-19T19:07:20.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.942+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a418f530 con 0x7f80a40ffbc0 2024-02-19T19:07:20.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.943+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f809400f5d0 con 0x7f80a40ffbc0 2024-02-19T19:07:20.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.943+0000 7f80a8a21700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f808c06d9e0 0x7f808c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:20.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.943+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f809408ea10 con 0x7f80a40ffbc0 2024-02-19T19:07:20.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.943+0000 7f80a27fc700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f808c06d9e0 0x7f808c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:20.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.943+0000 7f80a27fc700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f808c06d9e0 0x7f808c06fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8098007be0 tx=0x7f80980073d0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:20.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:20.948+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f809405bb20 con 0x7f80a40ffbc0 2024-02-19T19:07:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.198+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f80a402cc70 con 0x7f80a40ffbc0 2024-02-19T19:07:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.199+0000 7f80a8a21700 1 -- 172.21.15.39:0/4090519763 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f809405b6b0 con 0x7f80a40ffbc0 2024-02-19T19:07:21.219 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:07:21.219 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:07:21.220 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:07:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:21.222 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f808c06d9e0 msgr2=0x7f808c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f808c06d9e0 0x7f808c06fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8098007be0 tx=0x7f80980073d0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 msgr2=0x7f80a40699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40699d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f809400bda0 tx=0x7f809400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 shutdown_connections 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f808c06d9e0 0x7f808c06fea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80a40ffbc0 0x7f80a40699d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 --2- 172.21.15.39:0/4090519763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80a4100dc0 0x7f80a419b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 >> 172.21.15.39:0/4090519763 conn(0x7f80a40fb160 msgr2=0x7f80a41040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 shutdown_connections 2024-02-19T19:07:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.201+0000 7f80a9a23700 1 -- 172.21.15.39:0/4090519763 wait complete. 2024-02-19T19:07:21.224 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:07:21.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.320+0000 7f9d9c000700 1 -- 172.21.15.39:0/1655404536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94101740 msgr2=0x7f9d94103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.320+0000 7f9d9c000700 1 --2- 172.21.15.39:0/1655404536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94101740 0x7f9d94103b30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9d90009910 tx=0x7f9d90009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:21.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.321+0000 7f9d9c000700 1 -- 172.21.15.39:0/1655404536 shutdown_connections 2024-02-19T19:07:21.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.321+0000 7f9d9c000700 1 --2- 172.21.15.39:0/1655404536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94104070 0x7f9d94106460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.321+0000 7f9d9c000700 1 --2- 172.21.15.39:0/1655404536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94101740 0x7f9d94103b30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.321+0000 7f9d9c000700 1 -- 172.21.15.39:0/1655404536 >> 172.21.15.39:0/1655404536 conn(0x7f9d940fb150 msgr2=0x7f9d940fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.321+0000 7f9d9c000700 1 -- 172.21.15.39:0/1655404536 shutdown_connections 2024-02-19T19:07:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.321+0000 7f9d9c000700 1 -- 172.21.15.39:0/1655404536 wait complete. 2024-02-19T19:07:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9c000700 1 Processor -- start 2024-02-19T19:07:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9c000700 1 -- start start 2024-02-19T19:07:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9c000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94101740 0x7f9d941964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9c000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 0x7f9d941969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9c000700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d94196f70 con 0x7f9d94104070 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9c000700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941970b0 con 0x7f9d94101740 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9959b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 0x7f9d941969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9959b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 0x7f9d941969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58300/0 (socket says 172.21.15.39:58300) 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9959b700 1 -- 172.21.15.39:0/3542035249 learned_addr learned my addr 172.21.15.39:0/3542035249 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d99d9c700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94101740 0x7f9d941964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9959b700 1 -- 172.21.15.39:0/3542035249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94101740 msgr2=0x7f9d941964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9959b700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94101740 0x7f9d941964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.322+0000 7f9d9959b700 1 -- 172.21.15.39:0/3542035249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d900095f0 con 0x7f9d94104070 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d99d9c700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94101740 0x7f9d941964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d9959b700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 0x7f9d941969e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400c930 tx=0x7f9d8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d84007ad0 con 0x7f9d94104070 2024-02-19T19:07:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9d8400ce80 con 0x7f9d94104070 2024-02-19T19:07:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d84018630 con 0x7f9d94104070 2024-02-19T19:07:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9419bb70 con 0x7f9d94104070 2024-02-19T19:07:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.323+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9419c090 con 0x7f9d94104070 2024-02-19T19:07:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.324+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d94190620 con 0x7f9d94104070 2024-02-19T19:07:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.325+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f9d84007c30 con 0x7f9d94104070 2024-02-19T19:07:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.325+0000 7f9d8affd700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d8006d9e0 0x7f9d8006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.325+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9d8408d670 con 0x7f9d94104070 2024-02-19T19:07:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.325+0000 7f9d99d9c700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d8006d9e0 0x7f9d8006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.326+0000 7f9d99d9c700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d8006d9e0 0x7f9d8006fea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000b5c0 tx=0x7f9d90005690 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:21.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.330+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d8405a800 con 0x7f9d94104070 2024-02-19T19:07:21.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: pgmap v36: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:21.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:21.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:21.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/681766254' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/4090519763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:07:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.563+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d940611d0 con 0x7f9d8006d9e0 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.565+0000 7f9d8affd700 1 -- 172.21.15.39:0/3542035249 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f9d940611d0 con 0x7f9d8006d9e0 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: "mon" 2024-02-19T19:07:21.585 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:07:21.586 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "4/23 daemons upgraded", 2024-02-19T19:07:21.586 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading crash daemons", 2024-02-19T19:07:21.586 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:07:21.586 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d8006d9e0 msgr2=0x7f9d8006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d8006d9e0 0x7f9d8006fea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000b5c0 tx=0x7f9d90005690 comp rx=0 tx=0).stop 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 msgr2=0x7f9d941969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 0x7f9d941969e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400c930 tx=0x7f9d8400cc40 comp rx=0 tx=0).stop 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 shutdown_connections 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d8006d9e0 0x7f9d8006fea0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d94101740 0x7f9d941964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 --2- 172.21.15.39:0/3542035249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d94104070 0x7f9d941969e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 >> 172.21.15.39:0/3542035249 conn(0x7f9d940fb150 msgr2=0x7f9d940fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 shutdown_connections 2024-02-19T19:07:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.567+0000 7f9d9c000700 1 -- 172.21.15.39:0/3542035249 wait complete. 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.673+0000 7f6e308a2700 1 -- 172.21.15.39:0/2980535912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c101790 msgr2=0x7f6e2c103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.673+0000 7f6e308a2700 1 --2- 172.21.15.39:0/2980535912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c101790 0x7f6e2c103b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009910 tx=0x7f6e1c009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 -- 172.21.15.39:0/2980535912 shutdown_connections 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 --2- 172.21.15.39:0/2980535912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c1064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 --2- 172.21.15.39:0/2980535912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c101790 0x7f6e2c103b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 -- 172.21.15.39:0/2980535912 >> 172.21.15.39:0/2980535912 conn(0x7f6e2c0fb120 msgr2=0x7f6e2c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:21.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 -- 172.21.15.39:0/2980535912 shutdown_connections 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 -- 172.21.15.39:0/2980535912 wait complete. 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 Processor -- start 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.674+0000 7f6e308a2700 1 -- start start 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e308a2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c101790 0x7f6e2c1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e308a2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c0ff740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e308a2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c19a0e0 con 0x7f6e2c1040c0 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e308a2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c19a220 con 0x7f6e2c101790 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e29d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c0ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e29d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c0ff740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58322/0 (socket says 172.21.15.39:58322) 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e29d9b700 1 -- 172.21.15.39:0/1169463722 learned_addr learned my addr 172.21.15.39:0/1169463722 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:21.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e2a59c700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c101790 0x7f6e2c1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e2a59c700 1 -- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 msgr2=0x7f6e2c0ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e2a59c700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c0ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e2a59c700 1 -- 172.21.15.39:0/1169463722 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0095f0 con 0x7f6e2c101790 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e29d9b700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c0ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e2a59c700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c101790 0x7f6e2c1010f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009910 tx=0x7f6e1c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.675+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1c01d070 con 0x7f6e2c101790 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.676+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e1c005dc0 con 0x7f6e2c101790 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.676+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c0ffc80 con 0x7f6e2c101790 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.676+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1c017400 con 0x7f6e2c101790 2024-02-19T19:07:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.676+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c100140 con 0x7f6e2c101790 2024-02-19T19:07:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.676+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c190610 con 0x7f6e2c101790 2024-02-19T19:07:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.677+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 37) v1 ==== 91595+0+0 (secure 0 0 0) 0x7f6e1c00fac0 con 0x7f6e2c101790 2024-02-19T19:07:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.677+0000 7f6e1b7fe700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6e1406d9d0 0x7f6e1406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.678+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6e1c08ea80 con 0x7f6e2c101790 2024-02-19T19:07:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.678+0000 7f6e29d9b700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6e1406d9d0 0x7f6e1406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.678+0000 7f6e29d9b700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6e1406d9d0 0x7f6e1406fe90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e20011550 tx=0x7f6e20009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:21.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.682+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e1c05bb70 con 0x7f6e2c101790 2024-02-19T19:07:21.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: pgmap v36: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:21.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:21.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='client.44127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:21.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/681766254' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/4090519763' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:07:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.974+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6e2c1003b0 con 0x7f6e2c101790 2024-02-19T19:07:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.974+0000 7f6e1b7fe700 1 -- 172.21.15.39:0/1169463722 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f6e1c05b700 con 0x7f6e2c101790 2024-02-19T19:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6e1406d9d0 msgr2=0x7f6e1406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6e1406d9d0 0x7f6e1406fe90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e20011550 tx=0x7f6e20009260 comp rx=0 tx=0).stop 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c101790 msgr2=0x7f6e2c1010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c101790 0x7f6e2c1010f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c009910 tx=0x7f6e1c00f690 comp rx=0 tx=0).stop 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 shutdown_connections 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6e1406d9d0 0x7f6e1406fe90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e2c101790 0x7f6e2c1010f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 --2- 172.21.15.39:0/1169463722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e2c1040c0 0x7f6e2c0ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 >> 172.21.15.39:0/1169463722 conn(0x7f6e2c0fb120 msgr2=0x7f6e2c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 shutdown_connections 2024-02-19T19:07:21.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:21.976+0000 7f6e308a2700 1 -- 172.21.15.39:0/1169463722 wait complete. 2024-02-19T19:07:22.674 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:22 smithi086 ceph-mon[82675]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:22.674 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:22 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/1169463722' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:22.710 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:22 smithi039 ceph-mon[92224]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:22.710 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:22 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/1169463722' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:23.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:23 smithi086 ceph-mon[82675]: pgmap v37: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:23.623 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:23 smithi086 ceph-mon[82675]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:23.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:23 smithi039 ceph-mon[92224]: pgmap v37: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:23.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:23 smithi039 ceph-mon[92224]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:25.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:25 smithi039 ceph-mon[92224]: pgmap v38: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:25.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:25.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:25.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:25 smithi086 ceph-mon[82675]: pgmap v38: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:25.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:25 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:25.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:25 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:26.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:26 smithi086 ceph-mon[82675]: pgmap v39: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:26.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:26 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:26.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:26 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:26.831 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:26 smithi039 ceph-mon[92224]: pgmap v39: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:26.832 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:26 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:26.832 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:26 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:27.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:27 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:27.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:27 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:27.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:27 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:27.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:27 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:28.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:28 smithi086 ceph-mon[82675]: pgmap v40: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:28.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:28 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:28.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:28 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:28.883 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:28 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:28.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:28 smithi039 ceph-mon[92224]: pgmap v40: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:28.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:28 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:28.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:28 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:28.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:28 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:30.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:30.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:30.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:30.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:31.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:30 smithi039 ceph-mon[92224]: pgmap v41: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:31.285 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:30 smithi086 ceph-mon[82675]: pgmap v41: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:32.473 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi039"}]: dispatch 2024-02-19T19:07:32.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi039"}]': finished 2024-02-19T19:07:32.475 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]: dispatch 2024-02-19T19:07:32.475 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]': finished 2024-02-19T19:07:32.475 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:07:32.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi039"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi039"}]': finished 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]: dispatch 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi086"}]': finished 2024-02-19T19:07:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:07:33.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: pgmap v42: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:33.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: Upgrade: Setting container_image for all crash 2024-02-19T19:07:33.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:07:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: Upgrade: osd.0 is safe to restart 2024-02-19T19:07:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:07:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: pgmap v42: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: Upgrade: Setting container_image for all crash 2024-02-19T19:07:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:07:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: Upgrade: osd.0 is safe to restart 2024-02-19T19:07:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:33.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:07:33.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:34.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:34 smithi039 ceph-mon[92224]: Upgrade: Updating osd.0 2024-02-19T19:07:34.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:34 smithi039 ceph-mon[92224]: Deploying daemon osd.0 on smithi039 2024-02-19T19:07:34.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:34 smithi086 ceph-mon[82675]: Upgrade: Updating osd.0 2024-02-19T19:07:34.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:34 smithi086 ceph-mon[82675]: Deploying daemon osd.0 on smithi039 2024-02-19T19:07:35.639 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:35 smithi039 ceph-mon[92224]: pgmap v43: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:35.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:35 smithi086 ceph-mon[82675]: pgmap v43: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:07:36.737 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:36 smithi039 ceph-mon[92224]: pgmap v44: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:36.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:36 smithi086 ceph-mon[82675]: pgmap v44: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:07:37.394 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:37 smithi039 systemd[1]: Stopping Ceph osd.0 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:07:37.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:37 smithi039 ceph-mon[92224]: osd.0 marked itself down and dead 2024-02-19T19:07:37.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:37 smithi039 ceph-mon[92224]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:07:37.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[47506]: 2024-02-19T19:07:37.378+0000 7fb1c37e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:07:37.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[47506]: 2024-02-19T19:07:37.378+0000 7fb1c37e0700 -1 osd.0 49 *** Got signal Terminated *** 2024-02-19T19:07:37.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[47506]: 2024-02-19T19:07:37.378+0000 7fb1c37e0700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:07:37.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:37 smithi086 ceph-mon[82675]: osd.0 marked itself down and dead 2024-02-19T19:07:37.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:37 smithi086 ceph-mon[82675]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:07:38.247 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:37 smithi039 podman[100819]: 2024-02-19 19:07:37.91926985 +0000 UTC m=+0.659091219 container died a841adabd463b144c6c6db3c9ef37ffaac5c0e7c25209485799321cc7fa96827 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-19T19:07:38.711 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:38 smithi039 ceph-mon[92224]: pgmap v46: 65 pgs: 9 stale+active+clean, 56 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-02-19T19:07:38.711 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:38 smithi039 ceph-mon[92224]: osdmap e50: 6 total, 5 up, 6 in 2024-02-19T19:07:38.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:38 smithi039 podman[100819]: 2024-02-19 19:07:38.447134366 +0000 UTC m=+1.186955738 container cleanup a841adabd463b144c6c6db3c9ef37ffaac5c0e7c25209485799321cc7fa96827 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240214, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-19T19:07:38.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:38 smithi039 bash[100819]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0 2024-02-19T19:07:38.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:38 smithi086 ceph-mon[82675]: pgmap v46: 65 pgs: 9 stale+active+clean, 56 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-02-19T19:07:38.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:38 smithi086 ceph-mon[82675]: osdmap e50: 6 total, 5 up, 6 in 2024-02-19T19:07:38.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:38 smithi039 podman[100846]: 2024-02-19 19:07:38.690650912 +0000 UTC m=+0.774769199 container remove a841adabd463b144c6c6db3c9ef37ffaac5c0e7c25209485799321cc7fa96827 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2) 2024-02-19T19:07:39.398 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:39 smithi039 podman[100943]: 2024-02-19 19:07:39.019313212 +0000 UTC m=+0.018181650 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:39.398 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:39 smithi039 podman[100943]: 2024-02-19 19:07:39.13368789 +0000 UTC m=+0.132556356 container create 44ace277d57fb4e3afb7ca779d9273413f252101dd0a2232822693ce06265887 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2) 2024-02-19T19:07:39.712 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:39 smithi039 ceph-mon[92224]: osdmap e51: 6 total, 5 up, 6 in 2024-02-19T19:07:39.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:39 smithi039 podman[100943]: 2024-02-19 19:07:39.378157125 +0000 UTC m=+0.377025560 container init 44ace277d57fb4e3afb7ca779d9273413f252101dd0a2232822693ce06265887 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-02-19T19:07:39.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:39 smithi039 podman[100943]: 2024-02-19 19:07:39.381724818 +0000 UTC m=+0.380593254 container start 44ace277d57fb4e3afb7ca779d9273413f252101dd0a2232822693ce06265887 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214) 2024-02-19T19:07:39.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:39 smithi039 podman[100943]: 2024-02-19 19:07:39.478008286 +0000 UTC m=+0.476876720 container attach 44ace277d57fb4e3afb7ca779d9273413f252101dd0a2232822693ce06265887 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-19T19:07:39.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:39 smithi086 ceph-mon[82675]: osdmap e51: 6 total, 5 up, 6 in 2024-02-19T19:07:39.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:39 smithi039 podman[100943]: 2024-02-19 19:07:39.692578951 +0000 UTC m=+0.691447393 container died 44ace277d57fb4e3afb7ca779d9273413f252101dd0a2232822693ce06265887 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-19T19:07:40.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:40 smithi039 ceph-mon[92224]: pgmap v48: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:40.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:40 smithi039 ceph-mon[92224]: Health check failed: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:40.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:40 smithi039 podman[100969]: 2024-02-19 19:07:40.574691768 +0000 UTC m=+0.877354304 container remove 44ace277d57fb4e3afb7ca779d9273413f252101dd0a2232822693ce06265887 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:07:40.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:40 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-19T19:07:40.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:40 smithi039 systemd[1]: Stopped Ceph osd.0 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:07:40.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:40 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.0.service: Consumed 4.991s CPU time. 2024-02-19T19:07:40.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:40 smithi086 ceph-mon[82675]: pgmap v48: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:40.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:40 smithi086 ceph-mon[82675]: Health check failed: Degraded data redundancy: 15/72 objects degraded (20.833%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:41.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:40 smithi039 systemd[1]: Starting Ceph osd.0 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:07:41.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 podman[101073]: 2024-02-19 19:07:40.934097842 +0000 UTC m=+0.017945924 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:41.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 podman[101073]: 2024-02-19 19:07:41.08903004 +0000 UTC m=+0.172878126 container create 528112915beecf1139879ef6cf69e3ce7ad97ace1169ad432ef8a2a3aa07acb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-19T19:07:41.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 podman[101073]: 2024-02-19 19:07:41.34602003 +0000 UTC m=+0.429868124 container init 528112915beecf1139879ef6cf69e3ce7ad97ace1169ad432ef8a2a3aa07acb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-19T19:07:41.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 podman[101073]: 2024-02-19 19:07:41.349551912 +0000 UTC m=+0.433400001 container start 528112915beecf1139879ef6cf69e3ce7ad97ace1169ad432ef8a2a3aa07acb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-19T19:07:41.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 podman[101073]: 2024-02-19 19:07:41.453869235 +0000 UTC m=+0.537717318 container attach 528112915beecf1139879ef6cf69e3ce7ad97ace1169ad432ef8a2a3aa07acb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-19T19:07:42.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:07:42.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:07:42.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:07:42.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 bash[101073]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate[101088]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-19T19:07:42.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:41 smithi039 podman[101073]: 2024-02-19 19:07:41.820240459 +0000 UTC m=+0.904088542 container died 528112915beecf1139879ef6cf69e3ce7ad97ace1169ad432ef8a2a3aa07acb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-19T19:07:42.747 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:42 smithi039 ceph-mon[92224]: pgmap v49: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:42.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:42 smithi039 podman[101218]: 2024-02-19 19:07:42.550903937 +0000 UTC m=+0.725771977 container remove 528112915beecf1139879ef6cf69e3ce7ad97ace1169ad432ef8a2a3aa07acb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-19T19:07:42.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:42 smithi086 ceph-mon[82675]: pgmap v49: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:43.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:42 smithi039 podman[101283]: 2024-02-19 19:07:42.667462213 +0000 UTC m=+0.017839847 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:43.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:42 smithi039 podman[101283]: 2024-02-19 19:07:42.793833358 +0000 UTC m=+0.144211022 container create 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-19T19:07:43.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:43 smithi039 podman[101283]: 2024-02-19 19:07:43.055225466 +0000 UTC m=+0.405603097 container init 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:07:43.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:43 smithi039 podman[101283]: 2024-02-19 19:07:43.058449285 +0000 UTC m=+0.408826915 container start 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:07:43.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:43 smithi039 ceph-osd[101301]: -- 172.21.15.39:0/1949584768 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aa7e6af1e0 con 0x55aa7e681000 2024-02-19T19:07:43.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:43 smithi039 bash[101283]: 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 2024-02-19T19:07:43.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:43 smithi039 systemd[1]: Started Ceph osd.0 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:07:43.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:07:43.608+0000 7f8ab486f7c0 -1 Falling back to public interface 2024-02-19T19:07:44.337 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:44.337 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:44.337 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:44.337 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:44.337 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:44.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:44.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:44.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:44.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:44.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:45.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:45 smithi039 ceph-mon[92224]: pgmap v50: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-02-19T19:07:45.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:45 smithi039 ceph-mon[92224]: mgrmap e38: smithi039.ydmpuh(active, since 91s), standbys: smithi086.huubbu 2024-02-19T19:07:45.517 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:45 smithi086 ceph-mon[82675]: pgmap v50: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 15/72 objects degraded (20.833%) 2024-02-19T19:07:45.517 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:45 smithi086 ceph-mon[82675]: mgrmap e38: smithi039.ydmpuh(active, since 91s), standbys: smithi086.huubbu 2024-02-19T19:07:47.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:47 smithi039 ceph-mon[92224]: pgmap v51: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 383 B/s wr, 1 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:47.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:47 smithi039 ceph-mon[92224]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T19:07:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:07:47.070+0000 7f8ab486f7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-19T19:07:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:07:47.103+0000 7f8ab486f7c0 -1 osd.0 49 log_to_monitors true 2024-02-19T19:07:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:47 smithi086 ceph-mon[82675]: pgmap v51: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 383 B/s wr, 1 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:47.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:47 smithi086 ceph-mon[82675]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T19:07:48.427 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:07:48.085+0000 7f8aa1cbf700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:07:48.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:48 smithi039 ceph-mon[92224]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T19:07:48.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:48 smithi039 ceph-mon[92224]: osdmap e52: 6 total, 5 up, 6 in 2024-02-19T19:07:48.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:48 smithi039 ceph-mon[92224]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T19:07:48.428 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:48 smithi039 ceph-mon[92224]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' 2024-02-19T19:07:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:48 smithi086 ceph-mon[82675]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T19:07:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:48 smithi086 ceph-mon[82675]: osdmap e52: 6 total, 5 up, 6 in 2024-02-19T19:07:48.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:48 smithi086 ceph-mon[82675]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T19:07:48.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:48 smithi086 ceph-mon[82675]: from='osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553]' entity='osd.0' 2024-02-19T19:07:49.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: pgmap v53: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 341 B/s wr, 1 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:49.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:49.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:49.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:07:49.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553] boot 2024-02-19T19:07:49.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: osdmap e53: 6 total, 6 up, 6 in 2024-02-19T19:07:49.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:49 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:07:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: pgmap v53: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 341 B/s wr, 1 op/s; 15/72 objects degraded (20.833%) 2024-02-19T19:07:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:07:49.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: osd.0 [v2:172.21.15.39:6802/558256553,v1:172.21.15.39:6803/558256553] boot 2024-02-19T19:07:49.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: osdmap e53: 6 total, 6 up, 6 in 2024-02-19T19:07:49.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:49 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:07:51.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:51 smithi039 ceph-mon[92224]: pgmap v55: 65 pgs: 25 peering, 2 active+undersized+degraded, 7 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s; 4/72 objects degraded (5.556%) 2024-02-19T19:07:51.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:51 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:51.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:51 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:51.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:51 smithi039 ceph-mon[92224]: osdmap e54: 6 total, 6 up, 6 in 2024-02-19T19:07:51.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:51 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:51 smithi086 ceph-mon[82675]: pgmap v55: 65 pgs: 25 peering, 2 active+undersized+degraded, 7 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s; 4/72 objects degraded (5.556%) 2024-02-19T19:07:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:51 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:51 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:51 smithi086 ceph-mon[82675]: osdmap e54: 6 total, 6 up, 6 in 2024-02-19T19:07:51.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:51 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:52.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 -- 172.21.15.39:0/2169404344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd740fef60 msgr2=0x7fbd740ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2169404344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd740fef60 0x7fbd740ff380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b0a0 tx=0x7fbd6c00b3b0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 -- 172.21.15.39:0/2169404344 shutdown_connections 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2169404344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd74101f40 0x7fbd741023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2169404344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd740fef60 0x7fbd740ff380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 -- 172.21.15.39:0/2169404344 >> 172.21.15.39:0/2169404344 conn(0x7fbd740fa8c0 msgr2=0x7fbd740fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 -- 172.21.15.39:0/2169404344 shutdown_connections 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.086+0000 7fbd7884a700 1 -- 172.21.15.39:0/2169404344 wait complete. 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7884a700 1 Processor -- start 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7884a700 1 -- start start 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7884a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd740fef60 0x7fbd741018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7884a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 0x7fbd740ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7884a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74100430 con 0x7fbd74101f40 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7884a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74100570 con 0x7fbd740fef60 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd71d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 0x7fbd740ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd71d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 0x7fbd740ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59690/0 (socket says 172.21.15.39:59690) 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd71d9b700 1 -- 172.21.15.39:0/2212310694 learned_addr learned my addr 172.21.15.39:0/2212310694 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.087+0000 7fbd7259c700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd740fef60 0x7fbd741018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd71d9b700 1 -- 172.21.15.39:0/2212310694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd740fef60 msgr2=0x7fbd741018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd71d9b700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd740fef60 0x7fbd741018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd71d9b700 1 -- 172.21.15.39:0/2212310694 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c009d00 con 0x7fbd74101f40 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd71d9b700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 0x7fbd740ffef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00d8d0 tx=0x7fbd5c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c0098a0 con 0x7fbd74101f40 2024-02-19T19:07:52.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd74100850 con 0x7fbd74101f40 2024-02-19T19:07:52.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbd5c010460 con 0x7fbd74101f40 2024-02-19T19:07:52.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c00f5f0 con 0x7fbd74101f40 2024-02-19T19:07:52.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.088+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7419e510 con 0x7fbd74101f40 2024-02-19T19:07:52.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.089+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd74109800 con 0x7fbd74101f40 2024-02-19T19:07:52.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.090+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fbd5c009a00 con 0x7fbd74101f40 2024-02-19T19:07:52.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.090+0000 7fbd6b7fe700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbd6006d8a0 0x7fbd6006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.090+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbd5c08d2c0 con 0x7fbd74101f40 2024-02-19T19:07:52.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.090+0000 7fbd7259c700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbd6006d8a0 0x7fbd6006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.091+0000 7fbd7259c700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbd6006d8a0 0x7fbd6006fd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b830 tx=0x7fbd6c0060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:52.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.095+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbd5c05a670 con 0x7fbd74101f40 2024-02-19T19:07:52.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.320+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd74106890 con 0x7fbd6006d8a0 2024-02-19T19:07:52.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.322+0000 7fbd6b7fe700 1 -- 172.21.15.39:0/2212310694 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbd74106890 con 0x7fbd6006d8a0 2024-02-19T19:07:52.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbd6006d8a0 msgr2=0x7fbd6006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbd6006d8a0 0x7fbd6006fd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b830 tx=0x7fbd6c0060c0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 msgr2=0x7fbd740ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 0x7fbd740ffef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00d8d0 tx=0x7fbd5c00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 shutdown_connections 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbd6006d8a0 0x7fbd6006fd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd740fef60 0x7fbd741018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 --2- 172.21.15.39:0/2212310694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74101f40 0x7fbd740ffef0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 >> 172.21.15.39:0/2212310694 conn(0x7fbd740fa8c0 msgr2=0x7fbd74105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 shutdown_connections 2024-02-19T19:07:52.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.324+0000 7fbd7884a700 1 -- 172.21.15.39:0/2212310694 wait complete. 2024-02-19T19:07:52.356 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 -- 172.21.15.39:0/2486415460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00101820 msgr2=0x7fda00103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2486415460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00101820 0x7fda00103c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4009910 tx=0x7fd9f4009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 -- 172.21.15.39:0/2486415460 shutdown_connections 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2486415460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda00104150 0x7fda00106540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2486415460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00101820 0x7fda00103c10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 -- 172.21.15.39:0/2486415460 >> 172.21.15.39:0/2486415460 conn(0x7fda000fb190 msgr2=0x7fda000fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 -- 172.21.15.39:0/2486415460 shutdown_connections 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.433+0000 7fda06d0b700 1 -- 172.21.15.39:0/2486415460 wait complete. 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda06d0b700 1 Processor -- start 2024-02-19T19:07:52.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda06d0b700 1 -- start start 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda06d0b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda00101820 0x7fda00196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda06d0b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 0x7fda001969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda06d0b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda00196fd0 con 0x7fda00104150 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda06d0b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda00197110 con 0x7fda00101820 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fd9fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 0x7fda001969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fd9fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 0x7fda001969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59720/0 (socket says 172.21.15.39:59720) 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fd9fffff700 1 -- 172.21.15.39:0/2690369893 learned_addr learned my addr 172.21.15.39:0/2690369893 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fda04aa7700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda00101820 0x7fda00196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fd9fffff700 1 -- 172.21.15.39:0/2690369893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda00101820 msgr2=0x7fda00196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fd9fffff700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda00101820 0x7fda00196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.434+0000 7fd9fffff700 1 -- 172.21.15.39:0/2690369893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40095f0 con 0x7fda00104150 2024-02-19T19:07:52.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.435+0000 7fd9fffff700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 0x7fda001969b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e920 tx=0x7fd9ec00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.435+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9ec009790 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.435+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd9ec00cba0 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.435+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0019bbc0 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.435+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9ec018630 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.435+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0019c110 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.436+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda00190610 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.436+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fd9ec00cd10 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.437+0000 7fd9fdffb700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd9f006d8a0 0x7fd9f006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.437+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd9ec014070 con 0x7fda00104150 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.437+0000 7fda04aa7700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd9f006d8a0 0x7fd9f006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.437+0000 7fda04aa7700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd9f006d8a0 0x7fd9f006fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400b6f0 tx=0x7fd9f4009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:52.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.442+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd9ec052d20 con 0x7fda00104150 2024-02-19T19:07:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.668+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda000611d0 con 0x7fd9f006d8a0 2024-02-19T19:07:52.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.669+0000 7fd9fdffb700 1 -- 172.21.15.39:0/2690369893 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fda000611d0 con 0x7fd9f006d8a0 2024-02-19T19:07:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd9f006d8a0 msgr2=0x7fd9f006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd9f006d8a0 0x7fd9f006fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400b6f0 tx=0x7fd9f4009f70 comp rx=0 tx=0).stop 2024-02-19T19:07:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 msgr2=0x7fda001969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 0x7fda001969b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e920 tx=0x7fd9ec00ece0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 shutdown_connections 2024-02-19T19:07:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd9f006d8a0 0x7fd9f006fd60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda00101820 0x7fda00196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 --2- 172.21.15.39:0/2690369893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda00104150 0x7fda001969b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.672+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 >> 172.21.15.39:0/2690369893 conn(0x7fda000fb190 msgr2=0x7fda000fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.673+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 shutdown_connections 2024-02-19T19:07:52.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.673+0000 7fda06d0b700 1 -- 172.21.15.39:0/2690369893 wait complete. 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 -- 172.21.15.39:0/2869411052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 msgr2=0x7fec100fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 --2- 172.21.15.39:0/2869411052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec100fffe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fec00009910 tx=0x7fec00009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 -- 172.21.15.39:0/2869411052 shutdown_connections 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 --2- 172.21.15.39:0/2869411052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 0x7fec10101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 --2- 172.21.15.39:0/2869411052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec100fffe0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 -- 172.21.15.39:0/2869411052 >> 172.21.15.39:0/2869411052 conn(0x7fec100fb160 msgr2=0x7fec100fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 -- 172.21.15.39:0/2869411052 shutdown_connections 2024-02-19T19:07:52.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.780+0000 7fec16868700 1 -- 172.21.15.39:0/2869411052 wait complete. 2024-02-19T19:07:52.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec16868700 1 Processor -- start 2024-02-19T19:07:52.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec16868700 1 -- start start 2024-02-19T19:07:52.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec16868700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec10069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec16868700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 0x7fec1019b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec16868700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1019bb30 con 0x7fec100ffbc0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec16868700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1019bca0 con 0x7fec10100dc0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec0ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec10069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec0ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec10069990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59736/0 (socket says 172.21.15.39:59736) 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.781+0000 7fec0ffff700 1 -- 172.21.15.39:0/4032920027 learned_addr learned my addr 172.21.15.39:0/4032920027 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0f7fe700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 0x7fec1019b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0ffff700 1 -- 172.21.15.39:0/4032920027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 msgr2=0x7fec1019b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0ffff700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 0x7fec1019b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0ffff700 1 -- 172.21.15.39:0/4032920027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec000095f0 con 0x7fec100ffbc0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0f7fe700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 0x7fec1019b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0ffff700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec10069990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fec0000bda0 tx=0x7fec0000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec0001d070 con 0x7fec100ffbc0 2024-02-19T19:07:52.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fec0000f460 con 0x7fec100ffbc0 2024-02-19T19:07:52.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1019bf20 con 0x7fec100ffbc0 2024-02-19T19:07:52.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec00021640 con 0x7fec100ffbc0 2024-02-19T19:07:52.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.782+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1019c3c0 con 0x7fec100ffbc0 2024-02-19T19:07:52.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.783+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1018f550 con 0x7fec100ffbc0 2024-02-19T19:07:52.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.784+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fec00003a60 con 0x7fec100ffbc0 2024-02-19T19:07:52.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.784+0000 7fec0d7fa700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7febf806d8a0 0x7febf806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:52.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.784+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fec0008e820 con 0x7fec100ffbc0 2024-02-19T19:07:52.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.784+0000 7fec0f7fe700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7febf806d8a0 0x7febf806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:52.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.785+0000 7fec0f7fe700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7febf806d8a0 0x7febf806fd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fec04005fd0 tx=0x7fec04005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:52.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:52.789+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec0005baf0 con 0x7fec100ffbc0 2024-02-19T19:07:53.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.005+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fec100611d0 con 0x7febf806d8a0 2024-02-19T19:07:53.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.016+0000 7fec0d7fa700 1 -- 172.21.15.39:0/4032920027 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fec100611d0 con 0x7febf806d8a0 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (9m) 4s ago 11m 25.3M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (11m) 4s ago 11m 7931k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (10m) 28s ago 10m 7780k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (38s) 4s ago 11m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (32s) 28s ago 10m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (9m) 4s ago 11m 86.2M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (5m) 4s ago 5m 26.0M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (5m) 4s ago 5m 15.0M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (5m) 28s ago 5m 29.1M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (5m) 28s ago 5m 16.7M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (2m) 4s ago 13m 497M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:07:53.038 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (2m) 28s ago 10m 448M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (108s) 4s ago 13m 43.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (85s) 28s ago 10m 34.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (11m) 4s ago 11m 14.1M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (10m) 28s ago 10m 15.7M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (10s) 4s ago 9m 12.0M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (9m) 4s ago 9m 51.2M 1216M 18.2.1 94eadd990e0a 7ecc1a001f1a 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (8m) 4s ago 8m 50.5M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (8m) 28s ago 8m 49.8M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (7m) 28s ago 7m 47.2M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (7m) 28s ago 7m 49.3M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:07:53.039 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (2m) 4s ago 11m 62.0M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7febf806d8a0 msgr2=0x7febf806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7febf806d8a0 0x7febf806fd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fec04005fd0 tx=0x7fec04005ee0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 msgr2=0x7fec10069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec10069990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fec0000bda0 tx=0x7fec0000be80 comp rx=0 tx=0).stop 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 shutdown_connections 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7febf806d8a0 0x7febf806fd60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec100ffbc0 0x7fec10069990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 --2- 172.21.15.39:0/4032920027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec10100dc0 0x7fec1019b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 >> 172.21.15.39:0/4032920027 conn(0x7fec100fb160 msgr2=0x7fec10104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 shutdown_connections 2024-02-19T19:07:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.019+0000 7fec16868700 1 -- 172.21.15.39:0/4032920027 wait complete. 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 -- 172.21.15.39:0/3670475167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 msgr2=0x7f42d0102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 --2- 172.21.15.39:0/3670475167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0102300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009910 tx=0x7f42c0009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 -- 172.21.15.39:0/3670475167 shutdown_connections 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 --2- 172.21.15.39:0/3670475167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0102300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 --2- 172.21.15.39:0/3670475167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42d0100ca0 0x7f42d01010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 -- 172.21.15.39:0/3670475167 >> 172.21.15.39:0/3670475167 conn(0x7f42d00fc260 msgr2=0x7f42d00fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 -- 172.21.15.39:0/3670475167 shutdown_connections 2024-02-19T19:07:53.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.127+0000 7f42d7c93700 1 -- 172.21.15.39:0/3670475167 wait complete. 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d7c93700 1 Processor -- start 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d7c93700 1 -- start start 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d7c93700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42d0100ca0 0x7f42d01964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d7c93700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d7c93700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0197040 con 0x7f42d0101ea0 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d7c93700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0197180 con 0x7f42d0100ca0 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d522e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d522e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59752/0 (socket says 172.21.15.39:59752) 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d522e700 1 -- 172.21.15.39:0/4086849127 learned_addr learned my addr 172.21.15.39:0/4086849127 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d5a2f700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42d0100ca0 0x7f42d01964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d522e700 1 -- 172.21.15.39:0/4086849127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42d0100ca0 msgr2=0x7f42d01964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d522e700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42d0100ca0 0x7f42d01964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.128+0000 7f42d522e700 1 -- 172.21.15.39:0/4086849127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c00095f0 con 0x7f42d0101ea0 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.129+0000 7f42d522e700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0196a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f42c000b5c0 tx=0x7f42c000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.129+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42c001d070 con 0x7f42d0101ea0 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.129+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f42c0022470 con 0x7f42d0101ea0 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.129+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42c000f690 con 0x7f42d0101ea0 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.129+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d019bbd0 con 0x7f42d0101ea0 2024-02-19T19:07:53.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.129+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d019c0c0 con 0x7f42d0101ea0 2024-02-19T19:07:53.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.130+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d0109800 con 0x7f42d0101ea0 2024-02-19T19:07:53.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.131+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f42c00225e0 con 0x7f42d0101ea0 2024-02-19T19:07:53.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.131+0000 7f42c6ffd700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f42bc06d8a0 0x7f42bc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.131+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f42c008ec30 con 0x7f42d0101ea0 2024-02-19T19:07:53.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.131+0000 7f42d5a2f700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f42bc06d8a0 0x7f42bc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.132+0000 7f42d5a2f700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f42bc06d8a0 0x7f42bc06fd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f42d0101d00 tx=0x7f42cc00a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:53.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.136+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f42c005bf60 con 0x7f42d0101ea0 2024-02-19T19:07:53.454 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:53 smithi039 ceph-mon[92224]: pgmap v57: 65 pgs: 25 peering, 2 active+undersized+degraded, 7 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s; 4/72 objects degraded (5.556%) 2024-02-19T19:07:53.454 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:53 smithi039 ceph-mon[92224]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:53.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.432+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f42d01099f0 con 0x7f42d0101ea0 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.433+0000 7f42c6ffd700 1 -- 172.21.15.39:0/4086849127 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f42c005baf0 con 0x7f42d0101ea0 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 5 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f42bc06d8a0 msgr2=0x7f42bc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f42bc06d8a0 0x7f42bc06fd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f42d0101d00 tx=0x7f42cc00a380 comp rx=0 tx=0).stop 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 msgr2=0x7f42d0196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0196a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f42c000b5c0 tx=0x7f42c000bd70 comp rx=0 tx=0).stop 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 shutdown_connections 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f42bc06d8a0 0x7f42bc06fd60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42d0100ca0 0x7f42d01964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 --2- 172.21.15.39:0/4086849127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42d0101ea0 0x7f42d0196a20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 >> 172.21.15.39:0/4086849127 conn(0x7f42d00fc260 msgr2=0x7f42d01050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:53.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.436+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 shutdown_connections 2024-02-19T19:07:53.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.437+0000 7f42d7c93700 1 -- 172.21.15.39:0/4086849127 wait complete. 2024-02-19T19:07:53.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:53 smithi086 ceph-mon[82675]: pgmap v57: 65 pgs: 25 peering, 2 active+undersized+degraded, 7 active+undersized, 31 active+clean; 591 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s; 4/72 objects degraded (5.556%) 2024-02-19T19:07:53.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:53 smithi086 ceph-mon[82675]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 -- 172.21.15.39:0/3991769736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 msgr2=0x7f91f4106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/3991769736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f4106460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009910 tx=0x7f91e4009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 -- 172.21.15.39:0/3991769736 shutdown_connections 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/3991769736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f4106460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/3991769736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f4101740 0x7f91f4103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 -- 172.21.15.39:0/3991769736 >> 172.21.15.39:0/3991769736 conn(0x7f91f40fb150 msgr2=0x7f91f40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 -- 172.21.15.39:0/3991769736 shutdown_connections 2024-02-19T19:07:53.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.554+0000 7f91fb6c0700 1 -- 172.21.15.39:0/3991769736 wait complete. 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91fb6c0700 1 Processor -- start 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91fb6c0700 1 -- start start 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91fb6c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f4101740 0x7f91f41010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91fb6c0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f40ff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91fb6c0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40ffc30 con 0x7f91f4104070 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91fb6c0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40ffd70 con 0x7f91f4101740 2024-02-19T19:07:53.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91f8c5b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f40ff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91f8c5b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f40ff6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59782/0 (socket says 172.21.15.39:59782) 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91f8c5b700 1 -- 172.21.15.39:0/4219871496 learned_addr learned my addr 172.21.15.39:0/4219871496 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.555+0000 7f91f945c700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f4101740 0x7f91f41010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91f8c5b700 1 -- 172.21.15.39:0/4219871496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f4101740 msgr2=0x7f91f41010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91f8c5b700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f4101740 0x7f91f41010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91f8c5b700 1 -- 172.21.15.39:0/4219871496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40095f0 con 0x7f91f4104070 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91f8c5b700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f40ff6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91e4000c00 tx=0x7f91e400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e401d070 con 0x7f91f4104070 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f40ffff0 con 0x7f91f4104070 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f91e4005dc0 con 0x7f91f4104070 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f41004e0 con 0x7f91f4104070 2024-02-19T19:07:53.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.556+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e40226a0 con 0x7f91f4104070 2024-02-19T19:07:53.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.557+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4190620 con 0x7f91f4104070 2024-02-19T19:07:53.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.558+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f91e4022800 con 0x7f91f4104070 2024-02-19T19:07:53.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.558+0000 7f91ea7fc700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f91e006d8a0 0x7f91e006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.558+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f91e408e8c0 con 0x7f91f4104070 2024-02-19T19:07:53.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.558+0000 7f91f945c700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f91e006d8a0 0x7f91e006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.559+0000 7f91f945c700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f91e006d8a0 0x7f91e006fd60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009c60 tx=0x7f91f0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:53.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.563+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f91e405bc70 con 0x7f91f4104070 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.816+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f91f402cc70 con 0x7f91f4104070 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.817+0000 7f91ea7fc700 1 -- 172.21.15.39:0/4219871496 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f91e405b800 con 0x7f91f4104070 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:07:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:07:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:07:53.841 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:53.842 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f91e006d8a0 msgr2=0x7f91e006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f91e006d8a0 0x7f91e006fd60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009c60 tx=0x7f91f0009260 comp rx=0 tx=0).stop 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 msgr2=0x7f91f40ff6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f40ff6f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91e4000c00 tx=0x7f91e400f690 comp rx=0 tx=0).stop 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 shutdown_connections 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f91e006d8a0 0x7f91e006fd60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f4101740 0x7f91f41010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 --2- 172.21.15.39:0/4219871496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91f4104070 0x7f91f40ff6f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 >> 172.21.15.39:0/4219871496 conn(0x7f91f40fb150 msgr2=0x7f91f40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 shutdown_connections 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.820+0000 7f91fb6c0700 1 -- 172.21.15.39:0/4219871496 wait complete. 2024-02-19T19:07:53.843 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 -- 172.21.15.39:0/867614444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100dc0 msgr2=0x7f8d20101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 --2- 172.21.15.39:0/867614444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100dc0 0x7f8d20101220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009910 tx=0x7f8d14009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 -- 172.21.15.39:0/867614444 shutdown_connections 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 --2- 172.21.15.39:0/867614444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100dc0 0x7f8d20101220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 --2- 172.21.15.39:0/867614444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d200ffbc0 0x7f8d200fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 -- 172.21.15.39:0/867614444 >> 172.21.15.39:0/867614444 conn(0x7f8d200fb160 msgr2=0x7f8d200fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 -- 172.21.15.39:0/867614444 shutdown_connections 2024-02-19T19:07:53.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.936+0000 7f8d248d6700 1 -- 172.21.15.39:0/867614444 wait complete. 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d248d6700 1 Processor -- start 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d248d6700 1 -- start start 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d248d6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 0x7f8d20069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d248d6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d20100dc0 0x7f8d2019b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d248d6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2019baf0 con 0x7f8d200ffbc0 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d248d6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2019bc30 con 0x7f8d20100dc0 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d1e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 0x7f8d20069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d1e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 0x7f8d20069970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59806/0 (socket says 172.21.15.39:59806) 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d1e59c700 1 -- 172.21.15.39:0/1549748768 learned_addr learned my addr 172.21.15.39:0/1549748768 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.937+0000 7f8d1dd9b700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d20100dc0 0x7f8d2019b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d1e59c700 1 -- 172.21.15.39:0/1549748768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d20100dc0 msgr2=0x7f8d2019b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d1e59c700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d20100dc0 0x7f8d2019b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d1e59c700 1 -- 172.21.15.39:0/1549748768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140095f0 con 0x7f8d200ffbc0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d1e59c700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 0x7f8d20069970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000d900 tx=0x7f8d1000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1000b920 con 0x7f8d200ffbc0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2019bf10 con 0x7f8d200ffbc0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d10010460 con 0x7f8d200ffbc0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d1000f5f0 con 0x7f8d200ffbc0 2024-02-19T19:07:53.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.938+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2019c430 con 0x7f8d200ffbc0 2024-02-19T19:07:53.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.939+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2018f530 con 0x7f8d200ffbc0 2024-02-19T19:07:53.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.940+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f8d1000ba80 con 0x7f8d200ffbc0 2024-02-19T19:07:53.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.940+0000 7f8d0f7fe700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8d0806d8a0 0x7f8d0806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:53.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.940+0000 7f8d1dd9b700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8d0806d8a0 0x7f8d0806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:53.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.941+0000 7f8d1dd9b700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8d0806d8a0 0x7f8d0806fd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d20069ef0 tx=0x7f8d1400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:53.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.941+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d1005ace0 con 0x7f8d200ffbc0 2024-02-19T19:07:53.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:53.946+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8d1005a3b0 con 0x7f8d200ffbc0 2024-02-19T19:07:54.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.186+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d200611d0 con 0x7f8d0806d8a0 2024-02-19T19:07:54.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.187+0000 7f8d0f7fe700 1 -- 172.21.15.39:0/1549748768 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f8d200611d0 con 0x7f8d0806d8a0 2024-02-19T19:07:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:07:54.208 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:07:54.208 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "7/23 daemons upgraded", 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:07:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:07:54.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8d0806d8a0 msgr2=0x7f8d0806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:54.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8d0806d8a0 0x7f8d0806fd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d20069ef0 tx=0x7f8d1400b540 comp rx=0 tx=0).stop 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 msgr2=0x7f8d20069970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 0x7f8d20069970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000d900 tx=0x7f8d1000dc10 comp rx=0 tx=0).stop 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 shutdown_connections 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8d0806d8a0 0x7f8d0806fd60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d200ffbc0 0x7f8d20069970 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 --2- 172.21.15.39:0/1549748768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d20100dc0 0x7f8d2019b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 >> 172.21.15.39:0/1549748768 conn(0x7f8d200fb160 msgr2=0x7f8d20104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 shutdown_connections 2024-02-19T19:07:54.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.189+0000 7f8d248d6700 1 -- 172.21.15.39:0/1549748768 wait complete. 2024-02-19T19:07:54.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.296+0000 7f51a9fd9700 1 -- 172.21.15.39:0/1233454417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 msgr2=0x7f51a40ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.296+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/1233454417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a40ff160 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5198009910 tx=0x7f5198009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.296+0000 7f51a9fd9700 1 -- 172.21.15.39:0/1233454417 shutdown_connections 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.296+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/1233454417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a40ff160 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.297+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/1233454417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51a4102280 0x7f51a41026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.297+0000 7f51a9fd9700 1 -- 172.21.15.39:0/1233454417 >> 172.21.15.39:0/1233454417 conn(0x7f51a40fa8c0 msgr2=0x7f51a40fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.297+0000 7f51a9fd9700 1 -- 172.21.15.39:0/1233454417 shutdown_connections 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.297+0000 7f51a9fd9700 1 -- 172.21.15.39:0/1233454417 wait complete. 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.297+0000 7f51a9fd9700 1 Processor -- start 2024-02-19T19:07:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.297+0000 7f51a9fd9700 1 -- start start 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a9fd9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a4196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a9fd9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51a4102280 0x7f51a4196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a9fd9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a41970b0 con 0x7f51a40fed00 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a9fd9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a41971f0 con 0x7f51a4102280 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a4196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a4196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59820/0 (socket says 172.21.15.39:59820) 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 -- 172.21.15.39:0/3991126981 learned_addr learned my addr 172.21.15.39:0/3991126981 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a2ffd700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51a4102280 0x7f51a4196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 -- 172.21.15.39:0/3991126981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51a4102280 msgr2=0x7f51a4196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51a4102280 0x7f51a4196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 -- 172.21.15.39:0/3991126981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51980095f0 con 0x7f51a40fed00 2024-02-19T19:07:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a37fe700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a4196550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f519400e8f0 tx=0x7f519400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5194009790 con 0x7f51a40fed00 2024-02-19T19:07:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f519400c810 con 0x7f51a40fed00 2024-02-19T19:07:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5194003d20 con 0x7f51a40fed00 2024-02-19T19:07:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.298+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a419bca0 con 0x7f51a40fed00 2024-02-19T19:07:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.299+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a419c1a0 con 0x7f51a40fed00 2024-02-19T19:07:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.299+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4109810 con 0x7f51a40fed00 2024-02-19T19:07:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.301+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f51940041f0 con 0x7f51a40fed00 2024-02-19T19:07:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.301+0000 7f51a0ff9700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f518c06d8a0 0x7f518c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.301+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5194014070 con 0x7f51a40fed00 2024-02-19T19:07:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.301+0000 7f51a2ffd700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f518c06d8a0 0x7f518c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.302+0000 7f51a2ffd700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f518c06d8a0 0x7f518c06fd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f519800b690 tx=0x7f5198005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:54.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.305+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f519405ac70 con 0x7f51a40fed00 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/4086849127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/4219871496' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:54.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:54.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:54.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:54.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/4086849127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:54.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/4219871496' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:07:54.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:54.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:54.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:54.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:54.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:54.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.595+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f51a419c840 con 0x7f51a40fed00 2024-02-19T19:07:54.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.595+0000 7f51a0ff9700 1 -- 172.21.15.39:0/3991126981 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f5194003e80 con 0x7f51a40fed00 2024-02-19T19:07:54.617 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:07:54.617 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:07:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f518c06d8a0 msgr2=0x7f518c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f518c06d8a0 0x7f518c06fd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f519800b690 tx=0x7f5198005a90 comp rx=0 tx=0).stop 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 msgr2=0x7f51a4196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a4196550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f519400e8f0 tx=0x7f519400ec00 comp rx=0 tx=0).stop 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 shutdown_connections 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f518c06d8a0 0x7f518c06fd60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51a40fed00 0x7f51a4196550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 --2- 172.21.15.39:0/3991126981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51a4102280 0x7f51a4196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.597+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 >> 172.21.15.39:0/3991126981 conn(0x7f51a40fa8c0 msgr2=0x7f51a4105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.598+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 shutdown_connections 2024-02-19T19:07:54.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:07:54.598+0000 7f51a9fd9700 1 -- 172.21.15.39:0/3991126981 wait complete. 2024-02-19T19:07:55.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: pgmap v58: 65 pgs: 25 peering, 40 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 987 B/s rd, 1 op/s 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded) 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:55.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:55 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3991126981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: pgmap v58: 65 pgs: 25 peering, 40 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 987 B/s rd, 1 op/s 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded) 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:07:55.437 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-02-19T19:07:55.438 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:55.438 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:55 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3991126981' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:57.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:57 smithi039 ceph-mon[92224]: pgmap v59: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-02-19T19:07:57.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:57 smithi086 ceph-mon[82675]: pgmap v59: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-02-19T19:07:59.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:59 smithi039 ceph-mon[92224]: pgmap v60: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-02-19T19:07:59.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:59.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:07:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:59.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:59 smithi086 ceph-mon[82675]: pgmap v60: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-02-19T19:07:59.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:07:59.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:07:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:01.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:01 smithi039 ceph-mon[92224]: pgmap v61: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:08:01.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:01 smithi086 ceph-mon[82675]: pgmap v61: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:08:03.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:03 smithi039 ceph-mon[92224]: pgmap v62: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 901 B/s rd, 1 op/s 2024-02-19T19:08:03.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:03 smithi086 ceph-mon[82675]: pgmap v62: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 901 B/s rd, 1 op/s 2024-02-19T19:08:04.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:04 smithi086 ceph-mon[82675]: pgmap v63: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:04.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:04.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:04 smithi039 ceph-mon[92224]: pgmap v63: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:04.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:06.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:06 smithi086 ceph-mon[82675]: pgmap v64: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:06.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:06 smithi039 ceph-mon[92224]: pgmap v64: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:08.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:08 smithi086 ceph-mon[82675]: pgmap v65: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:08.997 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:08 smithi039 ceph-mon[92224]: pgmap v65: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:09.907 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:08:09.907 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:09 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:08:09.907 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:09 smithi039 ceph-mon[92224]: Upgrade: osd.1 is safe to restart 2024-02-19T19:08:10.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:08:10.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:09 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:08:10.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:09 smithi086 ceph-mon[82675]: Upgrade: osd.1 is safe to restart 2024-02-19T19:08:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:10 smithi039 ceph-mon[92224]: pgmap v66: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:11.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:10 smithi086 ceph-mon[82675]: pgmap v66: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:12.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:12 smithi039 ceph-mon[92224]: Upgrade: Updating osd.1 2024-02-19T19:08:12.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:12 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:12.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:12 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:08:12.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:12 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:12.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:12 smithi039 ceph-mon[92224]: Deploying daemon osd.1 on smithi039 2024-02-19T19:08:12.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:12 smithi086 ceph-mon[82675]: Upgrade: Updating osd.1 2024-02-19T19:08:12.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:12 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:12.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:12 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:08:12.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:12 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:12.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:12 smithi086 ceph-mon[82675]: Deploying daemon osd.1 on smithi039 2024-02-19T19:08:13.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:13 smithi039 ceph-mon[92224]: pgmap v67: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:13.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:13 smithi086 ceph-mon[82675]: pgmap v67: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:14.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:14.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:14.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:14 smithi039 systemd[1]: Stopping Ceph osd.1 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:08:15.470 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:15 smithi039 ceph-mon[92224]: pgmap v68: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:15.470 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:15 smithi039 ceph-mon[92224]: osd.1 marked itself down and dead 2024-02-19T19:08:15.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[54808]: 2024-02-19T19:08:15.063+0000 7f510f8a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:08:15.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[54808]: 2024-02-19T19:08:15.063+0000 7f510f8a7700 -1 osd.1 54 *** Got signal Terminated *** 2024-02-19T19:08:15.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[54808]: 2024-02-19T19:08:15.063+0000 7f510f8a7700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:08:15.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:15 smithi086 ceph-mon[82675]: pgmap v68: 65 pgs: 65 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:08:15.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:15 smithi086 ceph-mon[82675]: osd.1 marked itself down and dead 2024-02-19T19:08:15.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:15 smithi039 podman[106684]: 2024-02-19 19:08:15.450183531 +0000 UTC m=+0.512122020 container died 7ecc1a001f1a228903837fc774c195e17ea766108e1795575f946e9fd52063bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, RELEASE=HEAD) 2024-02-19T19:08:16.110 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:15 smithi039 podman[106684]: 2024-02-19 19:08:15.863559193 +0000 UTC m=+0.925497674 container cleanup 7ecc1a001f1a228903837fc774c195e17ea766108e1795575f946e9fd52063bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-19T19:08:16.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:15 smithi039 bash[106684]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1 2024-02-19T19:08:16.447 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:16 smithi039 ceph-mon[92224]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:08:16.448 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:16 smithi039 ceph-mon[92224]: osdmap e55: 6 total, 5 up, 6 in 2024-02-19T19:08:16.448 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:16 smithi039 ceph-mon[92224]: osdmap e56: 6 total, 5 up, 6 in 2024-02-19T19:08:16.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:16 smithi039 podman[106701]: 2024-02-19 19:08:16.090564002 +0000 UTC m=+0.642934010 container remove 7ecc1a001f1a228903837fc774c195e17ea766108e1795575f946e9fd52063bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, org.label-schema.license=GPLv2, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T19:08:16.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:16 smithi086 ceph-mon[82675]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:08:16.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:16 smithi086 ceph-mon[82675]: osdmap e55: 6 total, 5 up, 6 in 2024-02-19T19:08:16.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:16 smithi086 ceph-mon[82675]: osdmap e56: 6 total, 5 up, 6 in 2024-02-19T19:08:16.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:16 smithi039 podman[106796]: 2024-02-19 19:08:16.330196934 +0000 UTC m=+0.019436472 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:16.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:16 smithi039 podman[106796]: 2024-02-19 19:08:16.478194334 +0000 UTC m=+0.167433874 container create af325a843ac65d4483861fdd909bf1f50534ee8d4917bd51accb23c612bf364c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-19T19:08:17.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:16 smithi039 podman[106796]: 2024-02-19 19:08:16.745015114 +0000 UTC m=+0.434254653 container init af325a843ac65d4483861fdd909bf1f50534ee8d4917bd51accb23c612bf364c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-02-19T19:08:17.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:16 smithi039 podman[106796]: 2024-02-19 19:08:16.748786614 +0000 UTC m=+0.438026160 container start af325a843ac65d4483861fdd909bf1f50534ee8d4917bd51accb23c612bf364c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-19T19:08:17.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:16 smithi039 podman[106796]: 2024-02-19 19:08:16.844493417 +0000 UTC m=+0.533732976 container attach af325a843ac65d4483861fdd909bf1f50534ee8d4917bd51accb23c612bf364c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:08:17.427 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:17 smithi039 ceph-mon[92224]: pgmap v70: 65 pgs: 12 stale+active+clean, 53 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:08:17.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:17 smithi039 podman[106796]: 2024-02-19 19:08:17.045961637 +0000 UTC m=+0.735201178 container died af325a843ac65d4483861fdd909bf1f50534ee8d4917bd51accb23c612bf364c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-19T19:08:17.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:17 smithi086 ceph-mon[82675]: pgmap v70: 65 pgs: 12 stale+active+clean, 53 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:08:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:17 smithi039 podman[106822]: 2024-02-19 19:08:17.707524271 +0000 UTC m=+0.656761914 container remove af325a843ac65d4483861fdd909bf1f50534ee8d4917bd51accb23c612bf364c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:08:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:17 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-19T19:08:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:17 smithi039 systemd[1]: Stopped Ceph osd.1 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:08:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:17 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.1.service: Consumed 5.545s CPU time. 2024-02-19T19:08:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:17 smithi039 systemd[1]: Starting Ceph osd.1 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:08:18.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 podman[106924]: 2024-02-19 19:08:18.057951347 +0000 UTC m=+0.017986306 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:18.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 podman[106924]: 2024-02-19 19:08:18.192167619 +0000 UTC m=+0.152202617 container create cd78be7349f243d64dcd1002d2380dc78a55a0a2fc0b10ad7c3d3991808a4e73 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-19T19:08:18.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 podman[106924]: 2024-02-19 19:08:18.436249643 +0000 UTC m=+0.396284600 container init cd78be7349f243d64dcd1002d2380dc78a55a0a2fc0b10ad7c3d3991808a4e73 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:18.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 podman[106924]: 2024-02-19 19:08:18.439720916 +0000 UTC m=+0.399755875 container start cd78be7349f243d64dcd1002d2380dc78a55a0a2fc0b10ad7c3d3991808a4e73 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:08:18.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 podman[106924]: 2024-02-19 19:08:18.570043442 +0000 UTC m=+0.530078399 container attach cd78be7349f243d64dcd1002d2380dc78a55a0a2fc0b10ad7c3d3991808a4e73 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-19T19:08:19.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:19 smithi039 ceph-mon[92224]: pgmap v72: 65 pgs: 9 stale+active+clean, 5 active+undersized, 51 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-02-19T19:08:19.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:08:19.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:08:19.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:08:19.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:08:19.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:08:19.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 bash[106924]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate[106939]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-19T19:08:19.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:18 smithi039 podman[106924]: 2024-02-19 19:08:18.921938676 +0000 UTC m=+0.881973637 container died cd78be7349f243d64dcd1002d2380dc78a55a0a2fc0b10ad7c3d3991808a4e73 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) 2024-02-19T19:08:19.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:19 smithi086 ceph-mon[82675]: pgmap v72: 65 pgs: 9 stale+active+clean, 5 active+undersized, 51 active+clean; 591 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 1 op/s 2024-02-19T19:08:19.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:19 smithi039 podman[107079]: 2024-02-19 19:08:19.566383378 +0000 UTC m=+0.639805607 container remove cd78be7349f243d64dcd1002d2380dc78a55a0a2fc0b10ad7c3d3991808a4e73 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:19.923 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:19 smithi039 podman[107145]: 2024-02-19 19:08:19.680885494 +0000 UTC m=+0.018294134 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:19.923 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:19 smithi039 podman[107145]: 2024-02-19 19:08:19.792228161 +0000 UTC m=+0.129636786 container create 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-02-19T19:08:20.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:20 smithi039 podman[107145]: 2024-02-19 19:08:20.078682828 +0000 UTC m=+0.416091459 container init 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:08:20.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:20 smithi039 podman[107145]: 2024-02-19 19:08:20.081845974 +0000 UTC m=+0.419254597 container start 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:20.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:20 smithi039 ceph-osd[107164]: -- 172.21.15.39:0/4058593361 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x565022c411e0 con 0x565022c12c00 2024-02-19T19:08:20.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:20 smithi039 ceph-mon[92224]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:20.498 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:20 smithi039 bash[107145]: 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb 2024-02-19T19:08:20.498 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:20 smithi039 systemd[1]: Started Ceph osd.1 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:08:20.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:20 smithi086 ceph-mon[82675]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:20.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:08:20.639+0000 7f43c25d07c0 -1 Falling back to public interface 2024-02-19T19:08:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:21 smithi039 ceph-mon[92224]: pgmap v73: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:21.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:21.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:21 smithi086 ceph-mon[82675]: pgmap v73: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:21.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:21.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:21.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:23.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:23 smithi086 ceph-mon[82675]: pgmap v74: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:23.600 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:23 smithi039 ceph-mon[92224]: pgmap v74: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:24.413 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:24 smithi039 ceph-mon[92224]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T19:08:24.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:08:24.105+0000 7f43c25d07c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-19T19:08:24.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:08:24.133+0000 7f43c25d07c0 -1 osd.1 54 log_to_monitors true 2024-02-19T19:08:24.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:24 smithi086 ceph-mon[82675]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T19:08:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.751+0000 7f1e04930700 1 -- 172.21.15.39:0/3158731540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e00100e10 msgr2=0x7f1e00101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.751+0000 7f1e04930700 1 --2- 172.21.15.39:0/3158731540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e00100e10 0x7f1e00101270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009910 tx=0x7f1df4009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.751+0000 7f1e04930700 1 -- 172.21.15.39:0/3158731540 shutdown_connections 2024-02-19T19:08:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.751+0000 7f1e04930700 1 --2- 172.21.15.39:0/3158731540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e00100e10 0x7f1e00101270 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.751+0000 7f1e04930700 1 --2- 172.21.15.39:0/3158731540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 0x7f1e00100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.751+0000 7f1e04930700 1 -- 172.21.15.39:0/3158731540 >> 172.21.15.39:0/3158731540 conn(0x7f1e000fb190 msgr2=0x7f1e000fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:24.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.752+0000 7f1e04930700 1 -- 172.21.15.39:0/3158731540 shutdown_connections 2024-02-19T19:08:24.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.752+0000 7f1e04930700 1 -- 172.21.15.39:0/3158731540 wait complete. 2024-02-19T19:08:24.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1e04930700 1 Processor -- start 2024-02-19T19:08:24.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1e04930700 1 -- start start 2024-02-19T19:08:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1e04930700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 0x7f1e0019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1e04930700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 0x7f1e001a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1e04930700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0019d770 con 0x7f1e0019d2f0 2024-02-19T19:08:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1e04930700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0019d8e0 con 0x7f1e000ffc10 2024-02-19T19:08:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1dfe59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 0x7f1e0019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1dfdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 0x7f1e001a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1dfdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 0x7f1e001a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37790/0 (socket says 172.21.15.39:37790) 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1dfdd9b700 1 -- 172.21.15.39:0/3597536878 learned_addr learned my addr 172.21.15.39:0/3597536878 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.753+0000 7f1dfe59c700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 0x7f1e0019cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:41208/0 (socket says 172.21.15.39:41208) 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.754+0000 7f1dfdd9b700 1 -- 172.21.15.39:0/3597536878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 msgr2=0x7f1e0019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.754+0000 7f1dfdd9b700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 0x7f1e0019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.754+0000 7f1dfdd9b700 1 -- 172.21.15.39:0/3597536878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40095f0 con 0x7f1e0019d2f0 2024-02-19T19:08:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.754+0000 7f1dfdd9b700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 0x7f1e001a2320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1df4000c00 tx=0x7f1df400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.755+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df401d070 con 0x7f1e0019d2f0 2024-02-19T19:08:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.755+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1df400f460 con 0x7f1e0019d2f0 2024-02-19T19:08:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.755+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1df4021620 con 0x7f1e0019d2f0 2024-02-19T19:08:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.755+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e001a2860 con 0x7f1e0019d2f0 2024-02-19T19:08:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.755+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e001a2c60 con 0x7f1e0019d2f0 2024-02-19T19:08:24.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.756+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e001075f0 con 0x7f1e0019d2f0 2024-02-19T19:08:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.757+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f1df400f5d0 con 0x7f1e0019d2f0 2024-02-19T19:08:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.758+0000 7f1def7fe700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1de806d8f0 0x7f1de806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.758+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1df408e640 con 0x7f1e0019d2f0 2024-02-19T19:08:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.758+0000 7f1dfe59c700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1de806d8f0 0x7f1de806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.759+0000 7f1dfe59c700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1de806d8f0 0x7f1de806fdb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e0019e290 tx=0x7f1df000a380 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:24.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:24.765+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1df4057ec0 con 0x7f1e0019d2f0 2024-02-19T19:08:25.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.053+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e00105760 con 0x7f1de806d8f0 2024-02-19T19:08:25.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.055+0000 7f1def7fe700 1 -- 172.21.15.39:0/3597536878 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1e00105760 con 0x7f1de806d8f0 2024-02-19T19:08:25.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1de806d8f0 msgr2=0x7f1de806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1de806d8f0 0x7f1de806fdb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e0019e290 tx=0x7f1df000a380 comp rx=0 tx=0).stop 2024-02-19T19:08:25.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 msgr2=0x7f1e001a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 0x7f1e001a2320 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1df4000c00 tx=0x7f1df400bd70 comp rx=0 tx=0).stop 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 shutdown_connections 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1de806d8f0 0x7f1de806fdb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e000ffc10 0x7f1e0019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 --2- 172.21.15.39:0/3597536878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e0019d2f0 0x7f1e001a2320 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.057+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 >> 172.21.15.39:0/3597536878 conn(0x7f1e000fb190 msgr2=0x7f1e00104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.058+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 shutdown_connections 2024-02-19T19:08:25.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.058+0000 7f1e04930700 1 -- 172.21.15.39:0/3597536878 wait complete. 2024-02-19T19:08:25.092 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:08:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 -- 172.21.15.39:0/597487224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 msgr2=0x7fbe240fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 --2- 172.21.15.39:0/597487224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe240fffc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009910 tx=0x7fbe14009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 -- 172.21.15.39:0/597487224 shutdown_connections 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 --2- 172.21.15.39:0/597487224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe24100da0 0x7fbe24101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 --2- 172.21.15.39:0/597487224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe240fffc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 -- 172.21.15.39:0/597487224 >> 172.21.15.39:0/597487224 conn(0x7fbe240fb120 msgr2=0x7fbe240fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 -- 172.21.15.39:0/597487224 shutdown_connections 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.172+0000 7fbe2913e700 1 -- 172.21.15.39:0/597487224 wait complete. 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2913e700 1 Processor -- start 2024-02-19T19:08:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2913e700 1 -- start start 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2913e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe241953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2913e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe24100da0 0x7fbe24195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2913e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe24195f30 con 0x7fbe240ffba0 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2913e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe24196070 con 0x7fbe24100da0 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe22d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe241953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe22d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe241953d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37820/0 (socket says 172.21.15.39:37820) 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe22d9d700 1 -- 172.21.15.39:0/3382430201 learned_addr learned my addr 172.21.15.39:0/3382430201 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.173+0000 7fbe2259c700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe24100da0 0x7fbe24195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe22d9d700 1 -- 172.21.15.39:0/3382430201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe24100da0 msgr2=0x7fbe24195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe22d9d700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe24100da0 0x7fbe24195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe22d9d700 1 -- 172.21.15.39:0/3382430201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe140095f0 con 0x7fbe240ffba0 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe22d9d700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe241953d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400bda0 tx=0x7fbe1400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1401d070 con 0x7fbe240ffba0 2024-02-19T19:08:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbe14022470 con 0x7fbe240ffba0 2024-02-19T19:08:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1400f650 con 0x7fbe240ffba0 2024-02-19T19:08:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2419aac0 con 0x7fbe240ffba0 2024-02-19T19:08:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.174+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2419af60 con 0x7fbe240ffba0 2024-02-19T19:08:25.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.175+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe24108700 con 0x7fbe240ffba0 2024-02-19T19:08:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.176+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fbe14004af0 con 0x7fbe240ffba0 2024-02-19T19:08:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.176+0000 7fbe0bfff700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbe0c06d8a0 0x7fbe0c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.176+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbe1408ebe0 con 0x7fbe240ffba0 2024-02-19T19:08:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.176+0000 7fbe2259c700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbe0c06d8a0 0x7fbe0c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.177+0000 7fbe2259c700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbe0c06d8a0 0x7fbe0c06fd60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe18005950 tx=0x7fbe1800a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:25.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.182+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe1405c0e0 con 0x7fbe240ffba0 2024-02-19T19:08:25.325 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:08:25.082+0000 7f43afa20700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:08:25.325 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:25 smithi039 ceph-mon[92224]: pgmap v75: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:25.326 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:25 smithi039 ceph-mon[92224]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T19:08:25.326 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:25 smithi039 ceph-mon[92224]: osdmap e57: 6 total, 5 up, 6 in 2024-02-19T19:08:25.326 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:25 smithi039 ceph-mon[92224]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T19:08:25.326 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:25 smithi039 ceph-mon[92224]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' 2024-02-19T19:08:25.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.407+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe2419b5f0 con 0x7fbe0c06d8a0 2024-02-19T19:08:25.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.408+0000 7fbe0bfff700 1 -- 172.21.15.39:0/3382430201 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbe2419b5f0 con 0x7fbe0c06d8a0 2024-02-19T19:08:25.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbe0c06d8a0 msgr2=0x7fbe0c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbe0c06d8a0 0x7fbe0c06fd60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbe18005950 tx=0x7fbe1800a300 comp rx=0 tx=0).stop 2024-02-19T19:08:25.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 msgr2=0x7fbe241953d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe241953d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400bda0 tx=0x7fbe1400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 shutdown_connections 2024-02-19T19:08:25.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbe0c06d8a0 0x7fbe0c06fd60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe240ffba0 0x7fbe241953d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 --2- 172.21.15.39:0/3382430201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe24100da0 0x7fbe24195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.410+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 >> 172.21.15.39:0/3382430201 conn(0x7fbe240fb120 msgr2=0x7fbe24103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:25.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.411+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 shutdown_connections 2024-02-19T19:08:25.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.411+0000 7fbe2913e700 1 -- 172.21.15.39:0/3382430201 wait complete. 2024-02-19T19:08:25.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:25 smithi086 ceph-mon[82675]: pgmap v75: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:25.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:25 smithi086 ceph-mon[82675]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T19:08:25.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:25 smithi086 ceph-mon[82675]: osdmap e57: 6 total, 5 up, 6 in 2024-02-19T19:08:25.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:25 smithi086 ceph-mon[82675]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T19:08:25.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:25 smithi086 ceph-mon[82675]: from='osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663]' entity='osd.1' 2024-02-19T19:08:25.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.516+0000 7f2679958700 1 -- 172.21.15.39:0/2049906607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 msgr2=0x7f2674102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.516+0000 7f2679958700 1 --2- 172.21.15.39:0/2049906607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674102310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f265c009910 tx=0x7f265c009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:25.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.516+0000 7f2679958700 1 -- 172.21.15.39:0/2049906607 shutdown_connections 2024-02-19T19:08:25.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.516+0000 7f2679958700 1 --2- 172.21.15.39:0/2049906607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674102310 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 --2- 172.21.15.39:0/2049906607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2674100cb0 0x7f26741010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 -- 172.21.15.39:0/2049906607 >> 172.21.15.39:0/2049906607 conn(0x7f26740fc230 msgr2=0x7f26740fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:25.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 -- 172.21.15.39:0/2049906607 shutdown_connections 2024-02-19T19:08:25.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 -- 172.21.15.39:0/2049906607 wait complete. 2024-02-19T19:08:25.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 Processor -- start 2024-02-19T19:08:25.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 -- start start 2024-02-19T19:08:25.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2674100cb0 0x7f2674074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26740736c0 con 0x7f2674101eb0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.517+0000 7f2679958700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2674073800 con 0x7f2674100cb0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37846/0 (socket says 172.21.15.39:37846) 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 -- 172.21.15.39:0/4214920063 learned_addr learned my addr 172.21.15.39:0/4214920063 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f2672ffd700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2674100cb0 0x7f2674074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 -- 172.21.15.39:0/4214920063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2674100cb0 msgr2=0x7f2674074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2674100cb0 0x7f2674074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 -- 172.21.15.39:0/4214920063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0095f0 con 0x7f2674101eb0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f26727fc700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674073180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f265c000c00 tx=0x7f265c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265c01d070 con 0x7f2674101eb0 2024-02-19T19:08:25.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2674073a80 con 0x7f2674101eb0 2024-02-19T19:08:25.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f265c005dc0 con 0x7f2674101eb0 2024-02-19T19:08:25.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f265c0226a0 con 0x7f2674101eb0 2024-02-19T19:08:25.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.518+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2674073f70 con 0x7f2674101eb0 2024-02-19T19:08:25.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.519+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2674109830 con 0x7f2674101eb0 2024-02-19T19:08:25.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.520+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f265c0038f0 con 0x7f2674101eb0 2024-02-19T19:08:25.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.520+0000 7f2678956700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f266006d8a0 0x7f266006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.520+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f265c08e9a0 con 0x7f2674101eb0 2024-02-19T19:08:25.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.520+0000 7f2672ffd700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f266006d8a0 0x7f266006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.521+0000 7f2672ffd700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f266006d8a0 0x7f266006fd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2674101d10 tx=0x7f266400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:25.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.526+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f265c05bcd0 con 0x7f2674101eb0 2024-02-19T19:08:25.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.770+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f267419e950 con 0x7f266006d8a0 2024-02-19T19:08:25.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.781+0000 7f2678956700 1 -- 172.21.15.39:0/4214920063 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f267419e950 con 0x7f266006d8a0 2024-02-19T19:08:25.803 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:08:25.803 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (10m) 36s ago 12m 25.3M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:08:25.803 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (12m) 36s ago 12m 7931k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:08:25.803 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (11m) 61s ago 11m 7780k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:08:25.803 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (71s) 36s ago 12m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (65s) 61s ago 11m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (10m) 36s ago 11m 86.2M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (6m) 36s ago 6m 26.0M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (6m) 36s ago 6m 15.0M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (6m) 61s ago 6m 29.1M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (6m) 61s ago 6m 16.7M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (3m) 36s ago 13m 497M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (2m) 61s ago 10m 448M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (2m) 36s ago 13m 43.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (118s) 61s ago 10m 34.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (12m) 36s ago 12m 14.1M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (11m) 61s ago 11m 15.7M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (43s) 36s ago 10m 12.0M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:08:25.804 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 starting - - - 1216M 2024-02-19T19:08:25.805 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (9m) 36s ago 9m 50.5M 1216M 18.2.1 94eadd990e0a 83b3d495e96d 2024-02-19T19:08:25.805 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (8m) 61s ago 8m 49.8M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:08:25.805 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (8m) 61s ago 8m 47.2M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:08:25.805 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (7m) 61s ago 7m 49.3M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:08:25.805 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (2m) 36s ago 11m 62.0M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:08:25.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f266006d8a0 msgr2=0x7f266006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f266006d8a0 0x7f266006fd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2674101d10 tx=0x7f266400b040 comp rx=0 tx=0).stop 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 msgr2=0x7f2674073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674073180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f265c000c00 tx=0x7f265c00f690 comp rx=0 tx=0).stop 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 shutdown_connections 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f266006d8a0 0x7f266006fd60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2674100cb0 0x7f2674074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 --2- 172.21.15.39:0/4214920063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2674101eb0 0x7f2674073180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 >> 172.21.15.39:0/4214920063 conn(0x7f26740fc230 msgr2=0x7f26741050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 shutdown_connections 2024-02-19T19:08:25.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.785+0000 7f2679958700 1 -- 172.21.15.39:0/4214920063 wait complete. 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 -- 172.21.15.39:0/921340975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 msgr2=0x7f9760101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 --2- 172.21.15.39:0/921340975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f9760101210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9750009910 tx=0x7f9750009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 -- 172.21.15.39:0/921340975 shutdown_connections 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 --2- 172.21.15.39:0/921340975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f9760101210 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 --2- 172.21.15.39:0/921340975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97600ffbb0 0x7f97600fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 -- 172.21.15.39:0/921340975 >> 172.21.15.39:0/921340975 conn(0x7f97600fb150 msgr2=0x7f97600fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:25.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 -- 172.21.15.39:0/921340975 shutdown_connections 2024-02-19T19:08:25.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.893+0000 7f976853b700 1 -- 172.21.15.39:0/921340975 wait complete. 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f976853b700 1 Processor -- start 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f976853b700 1 -- start start 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f976853b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97600ffbb0 0x7f9760069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f976853b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f976019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f976853b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760197010 con 0x7f9760100db0 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f976853b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976019dae0 con 0x7f97600ffbb0 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f9765ad6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f976019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f9765ad6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f976019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37868/0 (socket says 172.21.15.39:37868) 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f9765ad6700 1 -- 172.21.15.39:0/1621673166 learned_addr learned my addr 172.21.15.39:0/1621673166 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f97662d7700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97600ffbb0 0x7f9760069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f9765ad6700 1 -- 172.21.15.39:0/1621673166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97600ffbb0 msgr2=0x7f9760069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f9765ad6700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97600ffbb0 0x7f9760069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.894+0000 7f9765ad6700 1 -- 172.21.15.39:0/1621673166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500095f0 con 0x7f9760100db0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.895+0000 7f9765ad6700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f976019d5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f975000f690 tx=0x7f975000f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.895+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975001d070 con 0x7f9760100db0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.895+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9750005dc0 con 0x7f9760100db0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.895+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976019dd30 con 0x7f9760100db0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.895+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97500226a0 con 0x7f9760100db0 2024-02-19T19:08:25.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.895+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976019e220 con 0x7f9760100db0 2024-02-19T19:08:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.896+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9760108710 con 0x7f9760100db0 2024-02-19T19:08:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.896+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f975000fb20 con 0x7f9760100db0 2024-02-19T19:08:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.897+0000 7f97577fe700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f974c06d8a0 0x7f974c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.897+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f975008e980 con 0x7f9760100db0 2024-02-19T19:08:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.897+0000 7f97662d7700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f974c06d8a0 0x7f974c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.897+0000 7f97662d7700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f974c06d8a0 0x7f974c06fd60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9760100c10 tx=0x7f975c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:25.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:25.906+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f975005bcb0 con 0x7f9760100db0 2024-02-19T19:08:26.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.213+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9760108900 con 0x7f9760100db0 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.213+0000 7f97577fe700 1 -- 172.21.15.39:0/1621673166 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f975005b840 con 0x7f9760100db0 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:08:26.236 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.215+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f974c06d8a0 msgr2=0x7f974c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.215+0000 7f976853b700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f974c06d8a0 0x7f974c06fd60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9760100c10 tx=0x7f975c009260 comp rx=0 tx=0).stop 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.215+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 msgr2=0x7f976019d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.215+0000 7f976853b700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f976019d5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f975000f690 tx=0x7f975000f6c0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 shutdown_connections 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f974c06d8a0 0x7f974c06fd60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97600ffbb0 0x7f9760069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 --2- 172.21.15.39:0/1621673166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9760100db0 0x7f976019d5a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 >> 172.21.15.39:0/1621673166 conn(0x7f97600fb150 msgr2=0x7f9760103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:26.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 shutdown_connections 2024-02-19T19:08:26.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.216+0000 7f976853b700 1 -- 172.21.15.39:0/1621673166 wait complete. 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 -- 172.21.15.39:0/54036121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a80ffbb0 msgr2=0x7f77a80fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 --2- 172.21.15.39:0/54036121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a80ffbb0 0x7f77a80fffd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f779c009910 tx=0x7f779c009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 -- 172.21.15.39:0/54036121 shutdown_connections 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 --2- 172.21.15.39:0/54036121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a8100db0 0x7f77a8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 --2- 172.21.15.39:0/54036121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a80ffbb0 0x7f77a80fffd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 -- 172.21.15.39:0/54036121 >> 172.21.15.39:0/54036121 conn(0x7f77a80fb150 msgr2=0x7f77a80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:26.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 -- 172.21.15.39:0/54036121 shutdown_connections 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 -- 172.21.15.39:0/54036121 wait complete. 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 Processor -- start 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 -- start start 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80ffbb0 0x7f77a8069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.329+0000 7f77af1f8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 0x7f77a819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77af1f8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a81970e0 con 0x7f77a8100db0 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77af1f8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a819dbb0 con 0x7f77a80ffbb0 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 0x7f77a819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 0x7f77a819d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37886/0 (socket says 172.21.15.39:37886) 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 -- 172.21.15.39:0/133834582 learned_addr learned my addr 172.21.15.39:0/133834582 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:26.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77acf94700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80ffbb0 0x7f77a8069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:26.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 -- 172.21.15.39:0/133834582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80ffbb0 msgr2=0x7f77a8069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80ffbb0 0x7f77a8069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 -- 172.21.15.39:0/133834582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0095f0 con 0x7f77a8100db0 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77acf94700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80ffbb0 0x7f77a8069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a7fff700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 0x7f77a819d670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f779400e970 tx=0x7f779400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f779400cba0 con 0x7f77a8100db0 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f779400eec0 con 0x7f77a8100db0 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a819de60 con 0x7f77a8100db0 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.330+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7794005690 con 0x7f77a8100db0 2024-02-19T19:08:26.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.331+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a819e380 con 0x7f77a8100db0 2024-02-19T19:08:26.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.331+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a8108730 con 0x7f77a8100db0 2024-02-19T19:08:26.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.332+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f7794004750 con 0x7f77a8100db0 2024-02-19T19:08:26.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.332+0000 7f77a5ffb700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f779806d8a0 0x7f779806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:26.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.332+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7794014070 con 0x7f77a8100db0 2024-02-19T19:08:26.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.333+0000 7f77acf94700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f779806d8a0 0x7f779806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:26.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.333+0000 7f77acf94700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f779806d8a0 0x7f779806fd60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f779c00b6f0 tx=0x7f779c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:26.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.337+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f779405ab20 con 0x7f77a8100db0 2024-02-19T19:08:26.497 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663] boot 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: osdmap e58: 6 total, 6 up, 6 in 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:08:26.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:26 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/1621673166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:26.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:26.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:26.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:26.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:08:26.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: osd.1 [v2:172.21.15.39:6810/4048799663,v1:172.21.15.39:6811/4048799663] boot 2024-02-19T19:08:26.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: osdmap e58: 6 total, 6 up, 6 in 2024-02-19T19:08:26.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:08:26.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:26 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/1621673166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:26.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.590+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f77a819ea20 con 0x7f77a8100db0 2024-02-19T19:08:26.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.591+0000 7f77a5ffb700 1 -- 172.21.15.39:0/133834582 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f779405a6b0 con 0x7f77a8100db0 2024-02-19T19:08:26.612 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:08:26.612 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:08:26.613 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:08:26.614 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:26.615 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f779806d8a0 msgr2=0x7f779806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f779806d8a0 0x7f779806fd60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f779c00b6f0 tx=0x7f779c005940 comp rx=0 tx=0).stop 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 msgr2=0x7f77a819d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 0x7f77a819d670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f779400e970 tx=0x7f779400ec80 comp rx=0 tx=0).stop 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 shutdown_connections 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f779806d8a0 0x7f779806fd60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80ffbb0 0x7f77a8069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 --2- 172.21.15.39:0/133834582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f77a8100db0 0x7f77a819d670 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 >> 172.21.15.39:0/133834582 conn(0x7f77a80fb150 msgr2=0x7f77a8103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:26.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 shutdown_connections 2024-02-19T19:08:26.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.593+0000 7f77af1f8700 1 -- 172.21.15.39:0/133834582 wait complete. 2024-02-19T19:08:26.617 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 -- 172.21.15.39:0/180171580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 msgr2=0x7f51341010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 --2- 172.21.15.39:0/180171580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f51341010c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5128009910 tx=0x7f5128009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 -- 172.21.15.39:0/180171580 shutdown_connections 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 --2- 172.21.15.39:0/180171580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5134101ea0 0x7f5134102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 --2- 172.21.15.39:0/180171580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f51341010c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 -- 172.21.15.39:0/180171580 >> 172.21.15.39:0/180171580 conn(0x7f51340fc260 msgr2=0x7f51340fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 -- 172.21.15.39:0/180171580 shutdown_connections 2024-02-19T19:08:26.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.702+0000 7f513908c700 1 -- 172.21.15.39:0/180171580 wait complete. 2024-02-19T19:08:26.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513908c700 1 Processor -- start 2024-02-19T19:08:26.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513908c700 1 -- start start 2024-02-19T19:08:26.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513908c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f5134074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:26.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513908c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5134101ea0 0x7f5134073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:26.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513908c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51340736c0 con 0x7f5134100ca0 2024-02-19T19:08:26.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513908c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134073800 con 0x7f5134101ea0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f5132d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f5134074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f5132d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f5134074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51080/0 (socket says 172.21.15.39:51080) 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f5132d9d700 1 -- 172.21.15.39:0/1689667997 learned_addr learned my addr 172.21.15.39:0/1689667997 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.703+0000 7f513259c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5134101ea0 0x7f5134073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f5132d9d700 1 -- 172.21.15.39:0/1689667997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5134101ea0 msgr2=0x7f5134073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f5132d9d700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5134101ea0 0x7f5134073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f5132d9d700 1 -- 172.21.15.39:0/1689667997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51280095f0 con 0x7f5134100ca0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f5132d9d700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f5134074b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f512800bda0 tx=0x7f512800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512801d070 con 0x7f5134100ca0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5134073a80 con 0x7f5134100ca0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f512800f460 con 0x7f5134100ca0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5128021620 con 0x7f5134100ca0 2024-02-19T19:08:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.704+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5134073f70 con 0x7f5134100ca0 2024-02-19T19:08:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.705+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5134109800 con 0x7f5134100ca0 2024-02-19T19:08:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.706+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f512800f5d0 con 0x7f5134100ca0 2024-02-19T19:08:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.706+0000 7f511bfff700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f511c06d8a0 0x7f511c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.706+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f512808e7a0 con 0x7f5134100ca0 2024-02-19T19:08:26.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.706+0000 7f513259c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f511c06d8a0 0x7f511c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:26.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.707+0000 7f513259c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f511c06d8a0 0x7f511c06fd60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5124005950 tx=0x7f512400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:26.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.711+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f512805bad0 con 0x7f5134100ca0 2024-02-19T19:08:26.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.937+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f513419e9b0 con 0x7f511c06d8a0 2024-02-19T19:08:26.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.938+0000 7f511bfff700 1 -- 172.21.15.39:0/1689667997 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f513419e9b0 con 0x7f511c06d8a0 2024-02-19T19:08:26.960 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:08:26.960 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:08:26.960 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:08:26.960 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:08:26.960 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "8/23 daemons upgraded", 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:08:26.961 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f511c06d8a0 msgr2=0x7f511c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f511c06d8a0 0x7f511c06fd60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5124005950 tx=0x7f512400b410 comp rx=0 tx=0).stop 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 msgr2=0x7f5134074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f5134074b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f512800bda0 tx=0x7f512800bdd0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 shutdown_connections 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f511c06d8a0 0x7f511c06fd60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5134100ca0 0x7f5134074b30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 --2- 172.21.15.39:0/1689667997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5134101ea0 0x7f5134073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:26.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 >> 172.21.15.39:0/1689667997 conn(0x7f51340fc260 msgr2=0x7f51341050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:26.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 shutdown_connections 2024-02-19T19:08:26.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:26.941+0000 7f513908c700 1 -- 172.21.15.39:0/1689667997 wait complete. 2024-02-19T19:08:27.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/2567930524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 msgr2=0x7f9d88103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/2567930524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d88103b40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800b0a0 tx=0x7f9d7800b3b0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/2567930524 shutdown_connections 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/2567930524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 0x7f9d88106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/2567930524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d88103b40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/2567930524 >> 172.21.15.39:0/2567930524 conn(0x7f9d880fb160 msgr2=0x7f9d880fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/2567930524 shutdown_connections 2024-02-19T19:08:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/2567930524 wait complete. 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 Processor -- start 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.052+0000 7f9d8fcf2700 1 -- start start 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8fcf2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d881967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8fcf2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 0x7f9d88196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8fcf2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88197350 con 0x7f9d88101750 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8fcf2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88197490 con 0x7f9d88104080 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8da8e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d881967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8da8e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d881967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51092/0 (socket says 172.21.15.39:51092) 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8da8e700 1 -- 172.21.15.39:0/966947164 learned_addr learned my addr 172.21.15.39:0/966947164 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8d28d700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 0x7f9d88196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8d28d700 1 -- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 msgr2=0x7f9d881967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:27.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8d28d700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d881967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8d28d700 1 -- 172.21.15.39:0/966947164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78009d00 con 0x7f9d88104080 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8da8e700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d881967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d8d28d700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 0x7f9d88196d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00e970 tx=0x7f9d7c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.053+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d7c00c9d0 con 0x7f9d88104080 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.054+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9d7c010430 con 0x7f9d88104080 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.054+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8819bf40 con 0x7f9d88104080 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.054+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d7c018650 con 0x7f9d88104080 2024-02-19T19:08:27.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.054+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8819c460 con 0x7f9d88104080 2024-02-19T19:08:27.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.055+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d88190630 con 0x7f9d88104080 2024-02-19T19:08:27.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.055+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f9d7c00cb30 con 0x7f9d88104080 2024-02-19T19:08:27.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.055+0000 7f9d86ffd700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d7006d8a0 0x7f9d7006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:27.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.056+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9d7c014070 con 0x7f9d88104080 2024-02-19T19:08:27.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.056+0000 7f9d8da8e700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d7006d8a0 0x7f9d7006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:27.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.056+0000 7f9d8da8e700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d7006d8a0 0x7f9d7006fd60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d78000f80 tx=0x7f9d7800b700 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:27.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.060+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d7c052d20 con 0x7f9d88104080 2024-02-19T19:08:27.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.365+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9d88068af0 con 0x7f9d88104080 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.365+0000 7f9d86ffd700 1 -- 172.21.15.39:0/966947164 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+648 (secure 0 0 0) 0x7f9d7c05aa40 con 0x7f9d88104080 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-02-19T19:08:27.387 INFO:teuthology.orchestra.run.smithi039.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-02-19T19:08:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d7006d8a0 msgr2=0x7f9d7006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d7006d8a0 0x7f9d7006fd60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9d78000f80 tx=0x7f9d7800b700 comp rx=0 tx=0).stop 2024-02-19T19:08:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 msgr2=0x7f9d88196d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 0x7f9d88196d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00e970 tx=0x7f9d7c00ec80 comp rx=0 tx=0).stop 2024-02-19T19:08:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 shutdown_connections 2024-02-19T19:08:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9d7006d8a0 0x7f9d7006fd60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d88101750 0x7f9d881967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 --2- 172.21.15.39:0/966947164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d88104080 0x7f9d88196d30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:27.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 >> 172.21.15.39:0/966947164 conn(0x7f9d880fb160 msgr2=0x7f9d880fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:27.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.367+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 shutdown_connections 2024-02-19T19:08:27.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:27.368+0000 7f9d8fcf2700 1 -- 172.21.15.39:0/966947164 wait complete. 2024-02-19T19:08:27.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:27.535 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: pgmap v77: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:27.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:27.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/133834582' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:08:27.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:27.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:27.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:27 smithi086 ceph-mon[82675]: osdmap e59: 6 total, 6 up, 6 in 2024-02-19T19:08:27.667 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:27.667 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: pgmap v77: 65 pgs: 5 active+undersized+degraded, 29 active+undersized, 31 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:27.667 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:27.668 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/133834582' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:08:27.668 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:27.668 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:27.668 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:27 smithi039 ceph-mon[92224]: osdmap e59: 6 total, 6 up, 6 in 2024-02-19T19:08:28.711 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:28 smithi039 ceph-mon[92224]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:28.711 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:28 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/966947164' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:08:28.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:28 smithi086 ceph-mon[82675]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:28.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:28 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/966947164' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:08:29.520 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:29 smithi039 ceph-mon[92224]: pgmap v80: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:29.520 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:29.520 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:29.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:29 smithi086 ceph-mon[82675]: pgmap v80: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 591 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 10/72 objects degraded (13.889%) 2024-02-19T19:08:29.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:29.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:30.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:30 smithi039 ceph-mon[92224]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded) 2024-02-19T19:08:30.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:30 smithi086 ceph-mon[82675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded) 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: pgmap v81: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:31.594 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:31.595 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:31.595 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:31.595 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:31.595 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:31 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:08:31.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: pgmap v81: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:31 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:08:32.644 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:32 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:08:32.644 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:32 smithi039 ceph-mon[92224]: Upgrade: osd.2 is safe to restart 2024-02-19T19:08:32.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:32 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:08:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:32 smithi086 ceph-mon[82675]: Upgrade: osd.2 is safe to restart 2024-02-19T19:08:33.532 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:33 smithi039 ceph-mon[92224]: pgmap v82: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:33.532 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:33.532 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:08:33.532 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:33 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:33.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:33 smithi086 ceph-mon[82675]: pgmap v82: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:33.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:33.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:08:33.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:33 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:34.613 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:34 smithi039 ceph-mon[92224]: Upgrade: Updating osd.2 2024-02-19T19:08:34.613 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:34 smithi039 ceph-mon[92224]: Deploying daemon osd.2 on smithi039 2024-02-19T19:08:34.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:34 smithi086 ceph-mon[82675]: Upgrade: Updating osd.2 2024-02-19T19:08:34.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:34 smithi086 ceph-mon[82675]: Deploying daemon osd.2 on smithi039 2024-02-19T19:08:35.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:35 smithi039 ceph-mon[92224]: pgmap v83: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-02-19T19:08:35.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:35 smithi086 ceph-mon[82675]: pgmap v83: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-02-19T19:08:36.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:36 smithi039 systemd[1]: Stopping Ceph osd.2 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:08:36.832 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[62484]: 2024-02-19T19:08:36.479+0000 7f5c67d58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:08:36.832 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[62484]: 2024-02-19T19:08:36.479+0000 7f5c67d58700 -1 osd.2 59 *** Got signal Terminated *** 2024-02-19T19:08:36.832 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[62484]: 2024-02-19T19:08:36.479+0000 7f5c67d58700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:08:37.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:37 smithi039 ceph-mon[92224]: pgmap v84: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 984 B/s rd, 1 op/s 2024-02-19T19:08:37.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:37 smithi039 ceph-mon[92224]: osd.2 marked itself down and dead 2024-02-19T19:08:37.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:37 smithi086 ceph-mon[82675]: pgmap v84: 65 pgs: 65 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 984 B/s rd, 1 op/s 2024-02-19T19:08:37.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:37 smithi086 ceph-mon[82675]: osd.2 marked itself down and dead 2024-02-19T19:08:38.127 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:37 smithi039 podman[112253]: 2024-02-19 19:08:37.7947487 +0000 UTC m=+1.437377337 container died 83b3d495e96d818f6ad70fb7d0587034f4a23db925b59d398353640d23d6abd0 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1) 2024-02-19T19:08:38.383 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:38 smithi039 podman[112253]: 2024-02-19 19:08:38.213762811 +0000 UTC m=+1.856391437 container cleanup 83b3d495e96d818f6ad70fb7d0587034f4a23db925b59d398353640d23d6abd0 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, RELEASE=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:08:38.383 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:38 smithi039 bash[112253]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2 2024-02-19T19:08:38.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:38 smithi039 ceph-mon[92224]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:08:38.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:38 smithi039 ceph-mon[92224]: osdmap e60: 6 total, 5 up, 6 in 2024-02-19T19:08:38.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:38 smithi039 ceph-mon[92224]: osdmap e61: 6 total, 5 up, 6 in 2024-02-19T19:08:38.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:38 smithi039 podman[112270]: 2024-02-19 19:08:38.448766023 +0000 UTC m=+0.656326212 container remove 83b3d495e96d818f6ad70fb7d0587034f4a23db925b59d398353640d23d6abd0 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-02-19T19:08:38.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:38 smithi086 ceph-mon[82675]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:08:38.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:38 smithi086 ceph-mon[82675]: osdmap e60: 6 total, 5 up, 6 in 2024-02-19T19:08:38.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:38 smithi086 ceph-mon[82675]: osdmap e61: 6 total, 5 up, 6 in 2024-02-19T19:08:39.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:38 smithi039 podman[112365]: 2024-02-19 19:08:38.783863773 +0000 UTC m=+0.018181347 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:39.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:38 smithi039 podman[112365]: 2024-02-19 19:08:38.916802429 +0000 UTC m=+0.151119998 container create 204bc42310b04635d028b8bfd21a35716b3cfeb2387434e5841aab81e9a1db30 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-19T19:08:39.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:39 smithi039 podman[112365]: 2024-02-19 19:08:39.169665114 +0000 UTC m=+0.403982688 container init 204bc42310b04635d028b8bfd21a35716b3cfeb2387434e5841aab81e9a1db30 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:08:39.487 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:39 smithi039 ceph-mon[92224]: pgmap v86: 65 pgs: 9 stale+active+clean, 56 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-02-19T19:08:39.487 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:39.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:39 smithi039 podman[112365]: 2024-02-19 19:08:39.172835521 +0000 UTC m=+0.407153098 container start 204bc42310b04635d028b8bfd21a35716b3cfeb2387434e5841aab81e9a1db30 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:08:39.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:39 smithi039 podman[112365]: 2024-02-19 19:08:39.26982779 +0000 UTC m=+0.504145363 container attach 204bc42310b04635d028b8bfd21a35716b3cfeb2387434e5841aab81e9a1db30 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=) 2024-02-19T19:08:39.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:39 smithi039 podman[112365]: 2024-02-19 19:08:39.46661945 +0000 UTC m=+0.700937020 container died 204bc42310b04635d028b8bfd21a35716b3cfeb2387434e5841aab81e9a1db30 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:08:39.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:39 smithi086 ceph-mon[82675]: pgmap v86: 65 pgs: 9 stale+active+clean, 56 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 1 op/s 2024-02-19T19:08:39.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:40.436 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 podman[112390]: 2024-02-19 19:08:40.082612747 +0000 UTC m=+0.610554442 container remove 204bc42310b04635d028b8bfd21a35716b3cfeb2387434e5841aab81e9a1db30 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214) 2024-02-19T19:08:40.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-19T19:08:40.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 systemd[1]: Stopped Ceph osd.2 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:08:40.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.2.service: Consumed 5.166s CPU time. 2024-02-19T19:08:40.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 systemd[1]: Starting Ceph osd.2 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:08:40.740 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:40 smithi039 ceph-mon[92224]: pgmap v88: 65 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 43 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-02-19T19:08:40.740 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:40 smithi039 ceph-mon[92224]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 4 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:40.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 podman[112493]: 2024-02-19 19:08:40.442829977 +0000 UTC m=+0.019604766 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:40.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 podman[112493]: 2024-02-19 19:08:40.583694918 +0000 UTC m=+0.160469699 container create 89a70362f77f04b8e9fdb6bd60d51b49fb47893e3ed12478c3d6250dd21e4be1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-19T19:08:40.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:40 smithi086 ceph-mon[82675]: pgmap v88: 65 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 43 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-02-19T19:08:40.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:40 smithi086 ceph-mon[82675]: Health check failed: Degraded data redundancy: 7/72 objects degraded (9.722%), 4 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:40.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 podman[112493]: 2024-02-19 19:08:40.828418149 +0000 UTC m=+0.405192942 container init 89a70362f77f04b8e9fdb6bd60d51b49fb47893e3ed12478c3d6250dd21e4be1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-19T19:08:40.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 podman[112493]: 2024-02-19 19:08:40.831831066 +0000 UTC m=+0.408605856 container start 89a70362f77f04b8e9fdb6bd60d51b49fb47893e3ed12478c3d6250dd21e4be1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:40.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:40 smithi039 podman[112493]: 2024-02-19 19:08:40.927993746 +0000 UTC m=+0.504768542 container attach 89a70362f77f04b8e9fdb6bd60d51b49fb47893e3ed12478c3d6250dd21e4be1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-19T19:08:41.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:08:41.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:41.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 bash[112493]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate[112508]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-19T19:08:41.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 podman[112493]: 2024-02-19 19:08:41.304315155 +0000 UTC m=+0.881089937 container died 89a70362f77f04b8e9fdb6bd60d51b49fb47893e3ed12478c3d6250dd21e4be1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:42.362 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:41 smithi039 podman[112648]: 2024-02-19 19:08:41.998866463 +0000 UTC m=+0.689425977 container remove 89a70362f77f04b8e9fdb6bd60d51b49fb47893e3ed12478c3d6250dd21e4be1 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-02-19T19:08:42.362 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 podman[112711]: 2024-02-19 19:08:42.113434793 +0000 UTC m=+0.017997596 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:42.362 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 podman[112711]: 2024-02-19 19:08:42.233573314 +0000 UTC m=+0.138136159 container create 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-19T19:08:42.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:42 smithi039 ceph-mon[92224]: pgmap v89: 65 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 43 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-02-19T19:08:42.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 podman[112711]: 2024-02-19 19:08:42.44471407 +0000 UTC m=+0.349276882 container init 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:08:42.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 podman[112711]: 2024-02-19 19:08:42.447781805 +0000 UTC m=+0.352344607 container start 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-19T19:08:42.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 ceph-osd[112730]: -- 172.21.15.39:0/4229809119 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c20defd1e0 con 0x55c20decec00 2024-02-19T19:08:42.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 bash[112711]: 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f 2024-02-19T19:08:42.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 systemd[1]: Started Ceph osd.2 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:08:42.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:42 smithi086 ceph-mon[82675]: pgmap v89: 65 pgs: 4 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 43 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-02-19T19:08:43.420 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:08:42.992+0000 7f259e7227c0 -1 Falling back to public interface 2024-02-19T19:08:43.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:43.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:43.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:43.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:43.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:44.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:44.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:44 smithi039 ceph-mon[92224]: pgmap v90: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:45.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:44 smithi086 ceph-mon[82675]: pgmap v90: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:46.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:46 smithi039 ceph-mon[92224]: pgmap v91: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:46.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:46 smithi039 ceph-mon[92224]: from='osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T19:08:46.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:08:46.446+0000 7f259e7227c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-19T19:08:46.748 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:08:46.469+0000 7f259e7227c0 -1 osd.2 59 log_to_monitors true 2024-02-19T19:08:46.749 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:08:46.632+0000 7f258fb7a700 -1 osd.2 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:08:47.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:46 smithi086 ceph-mon[82675]: pgmap v91: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 591 KiB data, 172 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:47.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:46 smithi086 ceph-mon[82675]: from='osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T19:08:47.917 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:47 smithi039 ceph-mon[92224]: from='osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T19:08:47.917 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:47 smithi039 ceph-mon[92224]: osdmap e62: 6 total, 5 up, 6 in 2024-02-19T19:08:47.917 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:47 smithi039 ceph-mon[92224]: from='osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T19:08:48.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:47 smithi086 ceph-mon[82675]: from='osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T19:08:48.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:47 smithi086 ceph-mon[82675]: osdmap e62: 6 total, 5 up, 6 in 2024-02-19T19:08:48.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:47 smithi086 ceph-mon[82675]: from='osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: pgmap v93: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837] boot 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: osdmap e63: 6 total, 6 up, 6 in 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:48.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:49.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: pgmap v93: 65 pgs: 6 active+undersized+degraded, 21 active+undersized, 38 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-02-19T19:08:49.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:08:49.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: osd.2 [v2:172.21.15.39:6818/2401360837,v1:172.21.15.39:6819/2401360837] boot 2024-02-19T19:08:49.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: osdmap e63: 6 total, 6 up, 6 in 2024-02-19T19:08:49.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:08:49.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:49.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:49.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:49 smithi039 ceph-mon[92224]: osdmap e64: 6 total, 6 up, 6 in 2024-02-19T19:08:50.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:49 smithi086 ceph-mon[82675]: osdmap e64: 6 total, 6 up, 6 in 2024-02-19T19:08:50.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:50 smithi039 ceph-mon[92224]: pgmap v96: 65 pgs: 3 active+undersized+degraded, 11 active+undersized, 51 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 7/72 objects degraded (9.722%) 2024-02-19T19:08:50.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:50 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 7/72 objects degraded (9.722%), 3 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:50.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:50.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:51.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:50 smithi086 ceph-mon[82675]: pgmap v96: 65 pgs: 3 active+undersized+degraded, 11 active+undersized, 51 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 7/72 objects degraded (9.722%) 2024-02-19T19:08:51.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:50 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 7/72 objects degraded (9.722%), 3 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:51.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:51.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:52.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:52 smithi039 ceph-mon[92224]: pgmap v97: 65 pgs: 3 active+undersized+degraded, 11 active+undersized, 51 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 7/72 objects degraded (9.722%) 2024-02-19T19:08:53.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:52 smithi086 ceph-mon[82675]: pgmap v97: 65 pgs: 3 active+undersized+degraded, 11 active+undersized, 51 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 7/72 objects degraded (9.722%) 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: pgmap v98: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 748 B/s rd, 1 op/s 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:54.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:54.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:08:54.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:08:54.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:54 smithi039 ceph-mon[92224]: Upgrade: osd.3 is safe to restart 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: pgmap v98: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 748 B/s rd, 1 op/s 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:54.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:54.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:54.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:54.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:08:54.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:08:54.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:54 smithi086 ceph-mon[82675]: Upgrade: osd.3 is safe to restart 2024-02-19T19:08:55.738 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:55 smithi086 ceph-mon[82675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/72 objects degraded (9.722%), 3 pgs degraded) 2024-02-19T19:08:55.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:55 smithi086 ceph-mon[82675]: Upgrade: Updating osd.3 2024-02-19T19:08:55.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:55.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:08:55.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:55 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:55.739 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:55 smithi086 ceph-mon[82675]: Deploying daemon osd.3 on smithi086 2024-02-19T19:08:55.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:55 smithi039 ceph-mon[92224]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/72 objects degraded (9.722%), 3 pgs degraded) 2024-02-19T19:08:55.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:55 smithi039 ceph-mon[92224]: Upgrade: Updating osd.3 2024-02-19T19:08:55.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:55.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:08:55.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:55 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:55.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:55 smithi039 ceph-mon[92224]: Deploying daemon osd.3 on smithi086 2024-02-19T19:08:56.657 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:56 smithi086 ceph-mon[82675]: pgmap v99: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:56.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:56 smithi039 ceph-mon[92224]: pgmap v99: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:08:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 -- 172.21.15.39:0/1374968239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98680ff320 msgr2=0x7f98680ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 --2- 172.21.15.39:0/1374968239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98680ff320 0x7f98680ff740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f986400b0a0 tx=0x7f986400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 -- 172.21.15.39:0/1374968239 shutdown_connections 2024-02-19T19:08:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 --2- 172.21.15.39:0/1374968239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9868100520 0x7f9868100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 --2- 172.21.15.39:0/1374968239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98680ff320 0x7f98680ff740 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 -- 172.21.15.39:0/1374968239 >> 172.21.15.39:0/1374968239 conn(0x7f98680fa8c0 msgr2=0x7f98680fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 -- 172.21.15.39:0/1374968239 shutdown_connections 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.489+0000 7f986fd5e700 1 -- 172.21.15.39:0/1374968239 wait complete. 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986fd5e700 1 Processor -- start 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986fd5e700 1 -- start start 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986fd5e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98680ff320 0x7f98681022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986fd5e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 0x7f986819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986fd5e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98681970c0 con 0x7f9868100520 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986fd5e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986819db90 con 0x7f98680ff320 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986d2f9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 0x7f986819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986d2f9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 0x7f986819d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35752/0 (socket says 172.21.15.39:35752) 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986d2f9700 1 -- 172.21.15.39:0/3485803025 learned_addr learned my addr 172.21.15.39:0/3485803025 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.490+0000 7f986dafa700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98680ff320 0x7f98681022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f986d2f9700 1 -- 172.21.15.39:0/3485803025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98680ff320 msgr2=0x7f98681022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:57.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f986d2f9700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98680ff320 0x7f98681022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f986d2f9700 1 -- 172.21.15.39:0/3485803025 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864009d00 con 0x7f9868100520 2024-02-19T19:08:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f986d2f9700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 0x7f986819d650 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f985800e950 tx=0x7f985800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f985800cba0 con 0x7f9868100520 2024-02-19T19:08:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f985800eea0 con 0x7f9868100520 2024-02-19T19:08:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986819de40 con 0x7f9868100520 2024-02-19T19:08:57.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9858018650 con 0x7f9868100520 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.491+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986819e360 con 0x7f9868100520 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.492+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9868107e80 con 0x7f9868100520 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.493+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f985800cd00 con 0x7f9868100520 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.493+0000 7f985effd700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f985406d8a0 0x7f985406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.493+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9858014070 con 0x7f9868100520 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.493+0000 7f986dafa700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f985406d8a0 0x7f985406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:57.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.494+0000 7f986dafa700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f985406d8a0 0x7f985406fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f986400b780 tx=0x7f98640060c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:57.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.498+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9858052d20 con 0x7f9868100520 2024-02-19T19:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.723+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9868104860 con 0x7f985406d8a0 2024-02-19T19:08:57.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.724+0000 7f985effd700 1 -- 172.21.15.39:0/3485803025 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9868104860 con 0x7f985406d8a0 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f985406d8a0 msgr2=0x7f985406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f985406d8a0 0x7f985406fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f986400b780 tx=0x7f98640060c0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 msgr2=0x7f986819d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 0x7f986819d650 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f985800e950 tx=0x7f985800ec60 comp rx=0 tx=0).stop 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 shutdown_connections 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f985406d8a0 0x7f985406fd60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98680ff320 0x7f98681022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 --2- 172.21.15.39:0/3485803025 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868100520 0x7f986819d650 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 >> 172.21.15.39:0/3485803025 conn(0x7f98680fa8c0 msgr2=0x7f98680fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 shutdown_connections 2024-02-19T19:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.726+0000 7f986fd5e700 1 -- 172.21.15.39:0/3485803025 wait complete. 2024-02-19T19:08:57.757 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3023900667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc78100da0 msgr2=0x7efc78101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3023900667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc78100da0 0x7efc78101200 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc60009910 tx=0x7efc60009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3023900667 shutdown_connections 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3023900667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc78100da0 0x7efc78101200 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3023900667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc780ffba0 0x7efc780fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3023900667 >> 172.21.15.39:0/3023900667 conn(0x7efc780fb120 msgr2=0x7efc780fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:57.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3023900667 shutdown_connections 2024-02-19T19:08:57.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3023900667 wait complete. 2024-02-19T19:08:57.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.833+0000 7efc7d8f2700 1 Processor -- start 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc7d8f2700 1 -- start start 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc7d8f2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 0x7efc781986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc7d8f2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc78100da0 0x7efc78198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc7d8f2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc78199220 con 0x7efc780ffba0 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc7d8f2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc78199360 con 0x7efc78100da0 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 0x7efc781986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:57.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 0x7efc781986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35772/0 (socket says 172.21.15.39:35772) 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 -- 172.21.15.39:0/3382066199 learned_addr learned my addr 172.21.15.39:0/3382066199 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc767fc700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc78100da0 0x7efc78198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 -- 172.21.15.39:0/3382066199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc78100da0 msgr2=0x7efc78198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc78100da0 0x7efc78198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 -- 172.21.15.39:0/3382066199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc68009c40 con 0x7efc780ffba0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.834+0000 7efc76ffd700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 0x7efc781986c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c140 tx=0x7efc6800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.835+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc6800e050 con 0x7efc780ffba0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.835+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efc6800f040 con 0x7efc780ffba0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.835+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc68007ea0 con 0x7efc780ffba0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.835+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc600095f0 con 0x7efc780ffba0 2024-02-19T19:08:57.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.835+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7819e170 con 0x7efc780ffba0 2024-02-19T19:08:57.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.836+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc78108700 con 0x7efc780ffba0 2024-02-19T19:08:57.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.837+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7efc68007500 con 0x7efc780ffba0 2024-02-19T19:08:57.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.837+0000 7efc7c8f0700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc6406d8a0 0x7efc6406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:57.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.837+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efc6808d600 con 0x7efc780ffba0 2024-02-19T19:08:57.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.837+0000 7efc767fc700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc6406d8a0 0x7efc6406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:57.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.838+0000 7efc767fc700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc6406d8a0 0x7efc6406fd60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc60009fd0 tx=0x7efc60005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:57.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:57.842+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc6805a930 con 0x7efc780ffba0 2024-02-19T19:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.062+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc78069540 con 0x7efc6406d8a0 2024-02-19T19:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.063+0000 7efc7c8f0700 1 -- 172.21.15.39:0/3382066199 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7efc78069540 con 0x7efc6406d8a0 2024-02-19T19:08:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc6406d8a0 msgr2=0x7efc6406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc6406d8a0 0x7efc6406fd60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc60009fd0 tx=0x7efc60005fd0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 msgr2=0x7efc781986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 0x7efc781986c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c140 tx=0x7efc6800c450 comp rx=0 tx=0).stop 2024-02-19T19:08:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 shutdown_connections 2024-02-19T19:08:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc6406d8a0 0x7efc6406fd60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc780ffba0 0x7efc781986c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 --2- 172.21.15.39:0/3382066199 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc78100da0 0x7efc78198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 >> 172.21.15.39:0/3382066199 conn(0x7efc780fb120 msgr2=0x7efc78068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 shutdown_connections 2024-02-19T19:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.065+0000 7efc7d8f2700 1 -- 172.21.15.39:0/3382066199 wait complete. 2024-02-19T19:08:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/593117809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 msgr2=0x7fe8c4100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/593117809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4100030 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009910 tx=0x7fe8b4009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/593117809 shutdown_connections 2024-02-19T19:08:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/593117809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 0x7fe8c4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/593117809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4100030 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/593117809 >> 172.21.15.39:0/593117809 conn(0x7fe8c40fb190 msgr2=0x7fe8c40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/593117809 shutdown_connections 2024-02-19T19:08:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/593117809 wait complete. 2024-02-19T19:08:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.171+0000 7fe8cb9dd700 1 Processor -- start 2024-02-19T19:08:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8cb9dd700 1 -- start start 2024-02-19T19:08:58.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8cb9dd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8cb9dd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 0x7fe8c41021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8cb9dd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4102730 con 0x7fe8c40ffc10 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8cb9dd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4102870 con 0x7fe8c4100e10 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c9779700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c9779700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35800/0 (socket says 172.21.15.39:35800) 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c9779700 1 -- 172.21.15.39:0/60390249 learned_addr learned my addr 172.21.15.39:0/60390249 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c8f78700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 0x7fe8c41021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c8f78700 1 -- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 msgr2=0x7fe8c4103ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c8f78700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4103ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c8f78700 1 -- 172.21.15.39:0/60390249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b40095f0 con 0x7fe8c4100e10 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c9779700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:08:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.172+0000 7fe8c8f78700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 0x7fe8c41021f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800e970 tx=0x7fe8b800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.173+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8b800c9d0 con 0x7fe8c4100e10 2024-02-19T19:08:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.173+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe8b8010430 con 0x7fe8c4100e10 2024-02-19T19:08:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.173+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8b8018650 con 0x7fe8c4100e10 2024-02-19T19:08:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.173+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c4102b50 con 0x7fe8c4100e10 2024-02-19T19:08:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.173+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c419e530 con 0x7fe8c4100e10 2024-02-19T19:08:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.174+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c4108700 con 0x7fe8c4100e10 2024-02-19T19:08:58.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.174+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fe8b80105a0 con 0x7fe8c4100e10 2024-02-19T19:08:58.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.175+0000 7fe8c27fc700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe8ac06d8a0 0x7fe8ac06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.175+0000 7fe8c9779700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe8ac06d8a0 0x7fe8ac06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.175+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe8b8014070 con 0x7fe8c4100e10 2024-02-19T19:08:58.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.175+0000 7fe8c9779700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe8ac06d8a0 0x7fe8ac06fd60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009fd0 tx=0x7fe8b4005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:58.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.180+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe8b8052d20 con 0x7fe8c4100e10 2024-02-19T19:08:58.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.391+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8c4068c90 con 0x7fe8ac06d8a0 2024-02-19T19:08:58.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.402+0000 7fe8c27fc700 1 -- 172.21.15.39:0/60390249 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fe8c4068c90 con 0x7fe8ac06d8a0 2024-02-19T19:08:58.423 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:08:58.423 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (11m) 9s ago 12m 24.8M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:08:58.423 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (12m) 9s ago 12m 8388k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (11m) 94s ago 11m 7780k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (104s) 9s ago 12m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (98s) 94s ago 11m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (10m) 9s ago 12m 88.3M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (6m) 9s ago 6m 26.4M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (6m) 9s ago 6m 15.4M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (6m) 94s ago 6m 29.1M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (6m) 94s ago 6m 16.7M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (3m) 9s ago 14m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (3m) 94s ago 11m 448M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (2m) 9s ago 14m 46.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (2m) 94s ago 11m 34.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (12m) 9s ago 12m 15.6M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (11m) 94s ago 11m 15.7M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:08:58.424 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (75s) 9s ago 10m 51.9M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (38s) 9s ago 10m 51.3M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 546fbde6ffab 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (16s) 9s ago 9m 11.7M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 3d2286828d68 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (9m) 94s ago 9m 49.8M 2240M 18.2.1 94eadd990e0a 50efb256ae55 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (8m) 94s ago 8m 47.2M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (8m) 94s ago 8m 49.3M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (3m) 9s ago 12m 62.3M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:08:58.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe8ac06d8a0 msgr2=0x7fe8ac06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe8ac06d8a0 0x7fe8ac06fd60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009fd0 tx=0x7fe8b4005940 comp rx=0 tx=0).stop 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 msgr2=0x7fe8c41021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 0x7fe8c41021f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800e970 tx=0x7fe8b800ec80 comp rx=0 tx=0).stop 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 shutdown_connections 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe8ac06d8a0 0x7fe8ac06fd60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8c40ffc10 0x7fe8c4103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 --2- 172.21.15.39:0/60390249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8c4100e10 0x7fe8c41021f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.404+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 >> 172.21.15.39:0/60390249 conn(0x7fe8c40fb190 msgr2=0x7fe8c4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.405+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 shutdown_connections 2024-02-19T19:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.405+0000 7fe8cb9dd700 1 -- 172.21.15.39:0/60390249 wait complete. 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 -- 172.21.15.39:0/1081040037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 msgr2=0x7f25841004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 --2- 172.21.15.39:0/1081040037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f25841004e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2574009910 tx=0x7f2574009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 -- 172.21.15.39:0/1081040037 shutdown_connections 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 --2- 172.21.15.39:0/1081040037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 0x7f2584101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 --2- 172.21.15.39:0/1081040037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f25841004e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 -- 172.21.15.39:0/1081040037 >> 172.21.15.39:0/1081040037 conn(0x7f25840fa8c0 msgr2=0x7f25840fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 -- 172.21.15.39:0/1081040037 shutdown_connections 2024-02-19T19:08:58.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.511+0000 7f258a478700 1 -- 172.21.15.39:0/1081040037 wait complete. 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f258a478700 1 Processor -- start 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f258a478700 1 -- start start 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f258a478700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f2584103b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f258a478700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 0x7f25841021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f258a478700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584102700 con 0x7f25841000c0 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f258a478700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2584102840 con 0x7f2584100de0 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f2583fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f2584103b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f2583fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f2584103b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35832/0 (socket says 172.21.15.39:35832) 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f2583fff700 1 -- 172.21.15.39:0/1471217500 learned_addr learned my addr 172.21.15.39:0/1471217500 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.512+0000 7f25837fe700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 0x7f25841021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f2583fff700 1 -- 172.21.15.39:0/1471217500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 msgr2=0x7f25841021c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f2583fff700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 0x7f25841021c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f2583fff700 1 -- 172.21.15.39:0/1471217500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740095f0 con 0x7f25841000c0 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f25837fe700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 0x7f25841021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f2583fff700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f2584103b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2574000c00 tx=0x7f257400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f257401d070 con 0x7f25841000c0 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2574005dc0 con 0x7f25841000c0 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2584102ac0 con 0x7f25841000c0 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2574022470 con 0x7f25841000c0 2024-02-19T19:08:58.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.513+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2584102fb0 con 0x7f25841000c0 2024-02-19T19:08:58.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.514+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2584108750 con 0x7f25841000c0 2024-02-19T19:08:58.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.515+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f2574003810 con 0x7f25841000c0 2024-02-19T19:08:58.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.515+0000 7f25817fa700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f256c06d8a0 0x7f256c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.515+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f257408e8f0 con 0x7f25841000c0 2024-02-19T19:08:58.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.515+0000 7f25837fe700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f256c06d8a0 0x7f256c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.516+0000 7f25837fe700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f256c06d8a0 0x7f256c06fd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25840ffdf0 tx=0x7f2578009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:58.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.520+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f257405bca0 con 0x7f25841000c0 2024-02-19T19:08:58.786 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:08:58 smithi086 systemd[1]: Stopping Ceph osd.3 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:08:58.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-mon[82675]: pgmap v100: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-02-19T19:08:58.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-mon[82675]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:58.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-mon[82675]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:58.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:58.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:58.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.812+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2584108940 con 0x7f25841000c0 2024-02-19T19:08:58.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:58 smithi039 ceph-mon[92224]: pgmap v100: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 2 op/s 2024-02-19T19:08:58.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:58 smithi039 ceph-mon[92224]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:58.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:58 smithi039 ceph-mon[92224]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:58.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:58.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.812+0000 7f25817fa700 1 -- 172.21.15.39:0/1471217500 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f257405b830 con 0x7f25841000c0 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-02-19T19:08:58.834 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 3 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 7 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:08:58.835 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f256c06d8a0 msgr2=0x7f256c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f256c06d8a0 0x7f256c06fd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25840ffdf0 tx=0x7f2578009260 comp rx=0 tx=0).stop 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 msgr2=0x7f2584103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f2584103b70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2574000c00 tx=0x7f257400f690 comp rx=0 tx=0).stop 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 shutdown_connections 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f256c06d8a0 0x7f256c06fd60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25841000c0 0x7f2584103b70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 --2- 172.21.15.39:0/1471217500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2584100de0 0x7f25841021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.814+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 >> 172.21.15.39:0/1471217500 conn(0x7f25840fa8c0 msgr2=0x7f25840fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:58.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.815+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 shutdown_connections 2024-02-19T19:08:58.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.815+0000 7f258a478700 1 -- 172.21.15.39:0/1471217500 wait complete. 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 -- 172.21.15.39:0/3754142262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0074d60 msgr2=0x7f0bd00731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/3754142262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0074d60 0x7f0bd00731c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009910 tx=0x7f0bc4009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 -- 172.21.15.39:0/3754142262 shutdown_connections 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/3754142262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bd0073790 0x7f0bd0073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/3754142262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0074d60 0x7f0bd00731c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 -- 172.21.15.39:0/3754142262 >> 172.21.15.39:0/3754142262 conn(0x7f0bd00fc230 msgr2=0x7f0bd00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 -- 172.21.15.39:0/3754142262 shutdown_connections 2024-02-19T19:08:58.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 -- 172.21.15.39:0/3754142262 wait complete. 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.920+0000 7f0bd61aa700 1 Processor -- start 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bd61aa700 1 -- start start 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bd61aa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 0x7f0bd01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bd61aa700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bd0074d60 0x7f0bd0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bd61aa700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0199230 con 0x7f0bd0073790 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bd61aa700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0199370 con 0x7f0bd0074d60 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 0x7f0bd01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 0x7f0bd01986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35844/0 (socket says 172.21.15.39:35844) 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 -- 172.21.15.39:0/285276682 learned_addr learned my addr 172.21.15.39:0/285276682 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bceffd700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bd0074d60 0x7f0bd0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 -- 172.21.15.39:0/285276682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bd0074d60 msgr2=0x7f0bd0198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bd0074d60 0x7f0bd0198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 -- 172.21.15.39:0/285276682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40095f0 con 0x7f0bd0073790 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bcf7fe700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 0x7f0bd01986d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4006010 tx=0x7f0bc400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.921+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bc401d070 con 0x7f0bd0073790 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.922+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0bc400f460 con 0x7f0bd0073790 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.922+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bc4021640 con 0x7f0bd0073790 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.922+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd019ddc0 con 0x7f0bd0073790 2024-02-19T19:08:58.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.922+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd019e200 con 0x7f0bd0073790 2024-02-19T19:08:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.923+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd010ba00 con 0x7f0bd0073790 2024-02-19T19:08:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.923+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f0bc400fab0 con 0x7f0bd0073790 2024-02-19T19:08:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.924+0000 7f0bccff9700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0bb806d8a0 0x7f0bb806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.924+0000 7f0bceffd700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0bb806d8a0 0x7f0bb806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.924+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0bc408e740 con 0x7f0bd0073790 2024-02-19T19:08:58.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.924+0000 7f0bceffd700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0bb806d8a0 0x7f0bb806fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005950 tx=0x7f0bc000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:58.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:58.928+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0bc405ba70 con 0x7f0bd0073790 2024-02-19T19:08:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.183+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0bd019e920 con 0x7f0bd0073790 2024-02-19T19:08:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.183+0000 7f0bccff9700 1 -- 172.21.15.39:0/285276682 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f0bc405b600 con 0x7f0bd0073790 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:08:59.205 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:08:59.206 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:08:59.207 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:08:59.208 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:59.208 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:59.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0bb806d8a0 msgr2=0x7f0bb806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0bb806d8a0 0x7f0bb806fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005950 tx=0x7f0bc000b410 comp rx=0 tx=0).stop 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 msgr2=0x7f0bd01986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 0x7f0bd01986d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4006010 tx=0x7f0bc400bd70 comp rx=0 tx=0).stop 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 shutdown_connections 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0bb806d8a0 0x7f0bb806fd60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd0073790 0x7f0bd01986d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 --2- 172.21.15.39:0/285276682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0bd0074d60 0x7f0bd0198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.186+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 >> 172.21.15.39:0/285276682 conn(0x7f0bd00fc230 msgr2=0x7f0bd0106af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.187+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 shutdown_connections 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.187+0000 7f0bd61aa700 1 -- 172.21.15.39:0/285276682 wait complete. 2024-02-19T19:08:59.209 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:08:59.286 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[43584]: 2024-02-19T19:08:58.806+0000 7f63d12fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:08:59.286 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[43584]: 2024-02-19T19:08:58.806+0000 7f63d12fd700 -1 osd.3 64 *** Got signal Terminated *** 2024-02-19T19:08:59.286 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:08:58 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[43584]: 2024-02-19T19:08:58.806+0000 7f63d12fd700 -1 osd.3 64 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 -- 172.21.15.39:0/513907015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 msgr2=0x7fbba8100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/513907015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8100290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009910 tx=0x7fbb90009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 -- 172.21.15.39:0/513907015 shutdown_connections 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/513907015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8100290 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/513907015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbba81011b0 0x7fbba81015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 -- 172.21.15.39:0/513907015 >> 172.21.15.39:0/513907015 conn(0x7fbba80fa8c0 msgr2=0x7fbba80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 -- 172.21.15.39:0/513907015 shutdown_connections 2024-02-19T19:08:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 -- 172.21.15.39:0/513907015 wait complete. 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.295+0000 7fbbaed3f700 1 Processor -- start 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbaed3f700 1 -- start start 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbaed3f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbaed3f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbba81011b0 0x7fbba8102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbaed3f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8102740 con 0x7fbba80ffe10 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbaed3f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8102880 con 0x7fbba81011b0 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8103bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35862/0 (socket says 172.21.15.39:35862) 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 -- 172.21.15.39:0/4023050930 learned_addr learned my addr 172.21.15.39:0/4023050930 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbba7fff700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbba81011b0 0x7fbba8102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 -- 172.21.15.39:0/4023050930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbba81011b0 msgr2=0x7fbba8102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbba81011b0 0x7fbba8102200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 -- 172.21.15.39:0/4023050930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900095f0 con 0x7fbba80ffe10 2024-02-19T19:08:59.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbacadb700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8103bb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800efc0 tx=0x7fbb9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:59.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb98018070 con 0x7fbba80ffe10 2024-02-19T19:08:59.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbb9800f040 con 0x7fbba80ffe10 2024-02-19T19:08:59.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba8102b00 con 0x7fbba80ffe10 2024-02-19T19:08:59.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.296+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb98004490 con 0x7fbba80ffe10 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.297+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba8103050 con 0x7fbba80ffe10 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.298+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba8108710 con 0x7fbba80ffe10 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.298+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fbb98004710 con 0x7fbba80ffe10 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.299+0000 7fbba5ffb700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbb9406d8a0 0x7fbb9406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.299+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fbb9808d730 con 0x7fbba80ffe10 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.299+0000 7fbba7fff700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbb9406d8a0 0x7fbb9406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:59.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.299+0000 7fbba7fff700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbb9406d8a0 0x7fbb9406fd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009fd0 tx=0x7fbb90005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:59.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.303+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb9805aa60 con 0x7fbba80ffe10 2024-02-19T19:08:59.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.540+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbba80ff410 con 0x7fbb9406d8a0 2024-02-19T19:08:59.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.542+0000 7fbba5ffb700 1 -- 172.21.15.39:0/4023050930 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fbba80ff410 con 0x7fbb9406d8a0 2024-02-19T19:08:59.563 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "9/23 daemons upgraded", 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:08:59.564 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbb9406d8a0 msgr2=0x7fbb9406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbb9406d8a0 0x7fbb9406fd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009fd0 tx=0x7fbb90005940 comp rx=0 tx=0).stop 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 msgr2=0x7fbba8103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8103bb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800efc0 tx=0x7fbb9800c3a0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 shutdown_connections 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbb9406d8a0 0x7fbb9406fd60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbba80ffe10 0x7fbba8103bb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 --2- 172.21.15.39:0/4023050930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbba81011b0 0x7fbba8102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 >> 172.21.15.39:0/4023050930 conn(0x7fbba80fa8c0 msgr2=0x7fbba80fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:59.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 shutdown_connections 2024-02-19T19:08:59.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.546+0000 7fbbaed3f700 1 -- 172.21.15.39:0/4023050930 wait complete. 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 -- 172.21.15.39:0/3380070485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c100cb0 msgr2=0x7f6c1c1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 --2- 172.21.15.39:0/3380070485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c100cb0 0x7f6c1c1010d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009910 tx=0x7f6c10009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 -- 172.21.15.39:0/3380070485 shutdown_connections 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 --2- 172.21.15.39:0/3380070485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 --2- 172.21.15.39:0/3380070485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c100cb0 0x7f6c1c1010d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 -- 172.21.15.39:0/3380070485 >> 172.21.15.39:0/3380070485 conn(0x7f6c1c0fc230 msgr2=0x7f6c1c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 -- 172.21.15.39:0/3380070485 shutdown_connections 2024-02-19T19:08:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.652+0000 7f6c23049700 1 -- 172.21.15.39:0/3380070485 wait complete. 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c23049700 1 Processor -- start 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c23049700 1 -- start start 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c23049700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c1c100cb0 0x7f6c1c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c23049700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c23049700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c1991b0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c23049700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c1992f0 con 0x7f6c1c100cb0 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c1bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c1bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35880/0 (socket says 172.21.15.39:35880) 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c1bfff700 1 -- 172.21.15.39:0/3373932975 learned_addr learned my addr 172.21.15.39:0/3373932975 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.653+0000 7f6c20de5700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c1c100cb0 0x7f6c1c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c1bfff700 1 -- 172.21.15.39:0/3373932975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c1c100cb0 msgr2=0x7f6c1c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c1bfff700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c1c100cb0 0x7f6c1c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c1bfff700 1 -- 172.21.15.39:0/3373932975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c100095f0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c1bfff700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c198c20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c930 tx=0x7f6c0800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c08007ad0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c19ddb0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c0800ce80 con 0x7f6c1c101eb0 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c08018630 con 0x7f6c1c101eb0 2024-02-19T19:08:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.654+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c19e2d0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.655+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c04f2e0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.656+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f6c08007c30 con 0x7f6c1c101eb0 2024-02-19T19:08:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.656+0000 7f6c19ffb700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6c0c06d8a0 0x7f6c0c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.656+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6c0808d4c0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.656+0000 7f6c20de5700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6c0c06d8a0 0x7f6c0c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.656+0000 7f6c20de5700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6c0c06d8a0 0x7f6c0c06fd60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c10006010 tx=0x7f6c1000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.661+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c0805a7f0 con 0x7f6c1c101eb0 2024-02-19T19:08:59.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:59 smithi086 ceph-mon[82675]: from='client.44207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:59.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:59.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:59 smithi086 ceph-mon[82675]: osd.3 marked itself down and dead 2024-02-19T19:08:59.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:59 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/1471217500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:59.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:08:59 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/285276682' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:08:59.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.950+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6c1c19e940 con 0x7f6c1c101eb0 2024-02-19T19:08:59.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:59 smithi039 ceph-mon[92224]: from='client.44207 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:59.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:08:59.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:59 smithi039 ceph-mon[92224]: osd.3 marked itself down and dead 2024-02-19T19:08:59.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:59 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/1471217500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:59.972 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:08:59 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/285276682' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:08:59.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.951+0000 7f6c19ffb700 1 -- 172.21.15.39:0/3373932975 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7f6c0805a380 con 0x7f6c1c101eb0 2024-02-19T19:08:59.973 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-02-19T19:08:59.973 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:08:59.973 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:08:59.973 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-19T19:08:59.973 INFO:teuthology.orchestra.run.smithi039.stdout: osd.3 (root=default,host=smithi086) is down 2024-02-19T19:08:59.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6c0c06d8a0 msgr2=0x7f6c0c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6c0c06d8a0 0x7f6c0c06fd60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c10006010 tx=0x7f6c1000b540 comp rx=0 tx=0).stop 2024-02-19T19:08:59.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 msgr2=0x7f6c1c198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c198c20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c930 tx=0x7f6c0800cc40 comp rx=0 tx=0).stop 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 shutdown_connections 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6c0c06d8a0 0x7f6c0c06fd60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c1c100cb0 0x7f6c1c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 --2- 172.21.15.39:0/3373932975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c1c101eb0 0x7f6c1c198c20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 >> 172.21.15.39:0/3373932975 conn(0x7f6c1c0fc230 msgr2=0x7f6c1c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 shutdown_connections 2024-02-19T19:08:59.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:08:59.953+0000 7f6c23049700 1 -- 172.21.15.39:0/3373932975 wait complete. 2024-02-19T19:09:00.179 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:08:59 smithi086 podman[92166]: 2024-02-19 19:08:59.867325554 +0000 UTC m=+1.184092923 container died 50efb256ae55597d38dde4fecf810d83af43ae3f6593d96f50097ef04b3ac31a (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-02-19T19:09:00.479 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:00 smithi086 podman[92166]: 2024-02-19 19:09:00.249624917 +0000 UTC m=+1.566392287 container cleanup 50efb256ae55597d38dde4fecf810d83af43ae3f6593d96f50097ef04b3ac31a (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-19T19:09:00.479 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:00 smithi086 bash[92166]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3 2024-02-19T19:09:00.786 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:00 smithi086 podman[92184]: 2024-02-19 19:09:00.45918443 +0000 UTC m=+0.596514757 container remove 50efb256ae55597d38dde4fecf810d83af43ae3f6593d96f50097ef04b3ac31a (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1) 2024-02-19T19:09:00.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:00 smithi086 ceph-mon[82675]: pgmap v101: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-02-19T19:09:00.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:00 smithi086 ceph-mon[82675]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:09:00.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:00 smithi086 ceph-mon[82675]: osdmap e65: 6 total, 5 up, 6 in 2024-02-19T19:09:00.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:00 smithi086 ceph-mon[82675]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:00.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:00 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3373932975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:09:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:00 smithi039 ceph-mon[92224]: pgmap v101: 65 pgs: 65 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-02-19T19:09:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:00 smithi039 ceph-mon[92224]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:09:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:00 smithi039 ceph-mon[92224]: osdmap e65: 6 total, 5 up, 6 in 2024-02-19T19:09:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:00 smithi039 ceph-mon[92224]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:00 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3373932975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:09:01.167 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:00 smithi086 podman[92281]: 2024-02-19 19:09:00.766254244 +0000 UTC m=+0.018806627 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:09:01.167 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:00 smithi086 podman[92281]: 2024-02-19 19:09:00.885291487 +0000 UTC m=+0.137843912 container create 6ea89d5c092fab2e4ecf6d240f1c6a2487cc77690f7ee74a01ccca8694e08e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:09:01.167 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:01 smithi086 podman[92281]: 2024-02-19 19:09:01.146339453 +0000 UTC m=+0.398891836 container init 6ea89d5c092fab2e4ecf6d240f1c6a2487cc77690f7ee74a01ccca8694e08e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-19T19:09:01.464 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:01 smithi086 podman[92281]: 2024-02-19 19:09:01.149523256 +0000 UTC m=+0.402075655 container start 6ea89d5c092fab2e4ecf6d240f1c6a2487cc77690f7ee74a01ccca8694e08e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-19T19:09:01.464 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:01 smithi086 podman[92281]: 2024-02-19 19:09:01.246291735 +0000 UTC m=+0.498844130 container attach 6ea89d5c092fab2e4ecf6d240f1c6a2487cc77690f7ee74a01ccca8694e08e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-02-19T19:09:01.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:01 smithi086 ceph-mon[82675]: osdmap e66: 6 total, 5 up, 6 in 2024-02-19T19:09:01.786 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:01 smithi086 podman[92281]: 2024-02-19 19:09:01.443846008 +0000 UTC m=+0.696398385 container died 6ea89d5c092fab2e4ecf6d240f1c6a2487cc77690f7ee74a01ccca8694e08e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:09:01.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:01 smithi039 ceph-mon[92224]: osdmap e66: 6 total, 5 up, 6 in 2024-02-19T19:09:02.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 podman[92307]: 2024-02-19 19:09:02.108917973 +0000 UTC m=+0.659679233 container remove 6ea89d5c092fab2e4ecf6d240f1c6a2487cc77690f7ee74a01ccca8694e08e24 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-19T19:09:02.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-19T19:09:02.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 systemd[1]: Stopped Ceph osd.3 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:09:02.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.3.service: Consumed 5.639s CPU time. 2024-02-19T19:09:02.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 systemd[1]: Starting Ceph osd.3 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:09:02.815 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:02 smithi086 ceph-mon[82675]: pgmap v104: 65 pgs: 16 stale+active+clean, 49 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:09:02.815 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 podman[92411]: 2024-02-19 19:09:02.455775774 +0000 UTC m=+0.017739405 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:09:02.815 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 podman[92411]: 2024-02-19 19:09:02.56854989 +0000 UTC m=+0.130513556 container create 8d4c2628f99526ecac68df80f1e99d5ca42e72e7b473f5145253c101bfd330f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:09:02.815 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 podman[92411]: 2024-02-19 19:09:02.794790851 +0000 UTC m=+0.356754473 container init 8d4c2628f99526ecac68df80f1e99d5ca42e72e7b473f5145253c101bfd330f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-19T19:09:02.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:02 smithi039 ceph-mon[92224]: pgmap v104: 65 pgs: 16 stale+active+clean, 49 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:09:03.261 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 podman[92411]: 2024-02-19 19:09:02.798276969 +0000 UTC m=+0.360240598 container start 8d4c2628f99526ecac68df80f1e99d5ca42e72e7b473f5145253c101bfd330f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:09:03.261 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:02 smithi086 podman[92411]: 2024-02-19 19:09:02.902709034 +0000 UTC m=+0.464672656 container attach 8d4c2628f99526ecac68df80f1e99d5ca42e72e7b473f5145253c101bfd330f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:09:03.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:09:03.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:09:03.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:09:03.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 bash[92411]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate[92426]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-19T19:09:03.537 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 podman[92411]: 2024-02-19 19:09:03.264811123 +0000 UTC m=+0.826774745 container died 8d4c2628f99526ecac68df80f1e99d5ca42e72e7b473f5145253c101bfd330f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-19T19:09:03.919 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:03 smithi086 ceph-mon[82675]: Health check failed: Degraded data redundancy: 5/72 objects degraded (6.944%), 3 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:03.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:03 smithi039 ceph-mon[92224]: Health check failed: Degraded data redundancy: 5/72 objects degraded (6.944%), 3 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:04.260 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:03 smithi086 podman[92562]: 2024-02-19 19:09:03.898799112 +0000 UTC m=+0.628478624 container remove 8d4c2628f99526ecac68df80f1e99d5ca42e72e7b473f5145253c101bfd330f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-02-19T19:09:04.260 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 podman[92629]: 2024-02-19 19:09:04.012431965 +0000 UTC m=+0.017885627 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:09:04.260 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 podman[92629]: 2024-02-19 19:09:04.116497279 +0000 UTC m=+0.121950982 container create 37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214) 2024-02-19T19:09:04.622 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 podman[92629]: 2024-02-19 19:09:04.343918926 +0000 UTC m=+0.349372583 container init 37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-19T19:09:04.622 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 podman[92629]: 2024-02-19 19:09:04.347408988 +0000 UTC m=+0.352862654 container start 37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:09:04.623 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 ceph-osd[92648]: -- 172.21.15.86:0/2798718625 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580648e11e0 con 0x5580648b2c00 2024-02-19T19:09:04.623 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 bash[92629]: 37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f 2024-02-19T19:09:04.623 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 systemd[1]: Started Ceph osd.3 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:09:04.916 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:04 smithi086 ceph-mon[82675]: pgmap v105: 65 pgs: 3 active+undersized+degraded, 12 stale+active+clean, 8 active+undersized, 42 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-02-19T19:09:04.917 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:04.917 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:04.917 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:04.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:04 smithi039 ceph-mon[92224]: pgmap v105: 65 pgs: 3 active+undersized+degraded, 12 stale+active+clean, 8 active+undersized, 42 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-02-19T19:09:04.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:04.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:04.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:05.285 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:04 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[92644]: 2024-02-19T19:09:04.895+0000 7fa6aaa627c0 -1 Falling back to public interface 2024-02-19T19:09:06.899 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:06 smithi039 ceph-mon[92224]: pgmap v106: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-02-19T19:09:06.997 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:06 smithi086 ceph-mon[82675]: pgmap v106: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-02-19T19:09:08.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[92644]: 2024-02-19T19:09:08.377+0000 7fa6aaa627c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-19T19:09:08.536 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[92644]: 2024-02-19T19:09:08.406+0000 7fa6aaa627c0 -1 osd.3 64 log_to_monitors true 2024-02-19T19:09:08.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:08 smithi039 ceph-mon[92224]: pgmap v107: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-02-19T19:09:08.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:08 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:08.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:08 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:08.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:08 smithi039 ceph-mon[92224]: from='osd.3 [v2:172.21.15.86:6800/2382739218,v1:172.21.15.86:6801/2382739218]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:09:08.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:08 smithi039 ceph-mon[92224]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:09:09.035 INFO:journalctl@ceph.osd.3.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-3[92644]: 2024-02-19T19:09:08.649+0000 7fa69beba700 -1 osd.3 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:09:09.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-mon[82675]: pgmap v107: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-02-19T19:09:09.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:09.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:09.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-mon[82675]: from='osd.3 [v2:172.21.15.86:6800/2382739218,v1:172.21.15.86:6801/2382739218]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:09:09.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:08 smithi086 ceph-mon[82675]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:09:09.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:09 smithi039 ceph-mon[92224]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T19:09:09.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:09 smithi039 ceph-mon[92224]: osdmap e67: 6 total, 5 up, 6 in 2024-02-19T19:09:09.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:09 smithi039 ceph-mon[92224]: from='osd.3 [v2:172.21.15.86:6800/2382739218,v1:172.21.15.86:6801/2382739218]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:09.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:09 smithi039 ceph-mon[92224]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:09.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:09.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:09 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:10.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:09 smithi086 ceph-mon[82675]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T19:09:10.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:09 smithi086 ceph-mon[82675]: osdmap e67: 6 total, 5 up, 6 in 2024-02-19T19:09:10.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:09 smithi086 ceph-mon[82675]: from='osd.3 [v2:172.21.15.86:6800/2382739218,v1:172.21.15.86:6801/2382739218]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:10.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:09 smithi086 ceph-mon[82675]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:10.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:10.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:09 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:10.670 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:10 smithi086 ceph-mon[82675]: pgmap v109: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-02-19T19:09:10.670 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:10 smithi086 ceph-mon[82675]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:09:10.670 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:10 smithi086 ceph-mon[82675]: osd.3 [v2:172.21.15.86:6800/2382739218,v1:172.21.15.86:6801/2382739218] boot 2024-02-19T19:09:10.670 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:10 smithi086 ceph-mon[82675]: osdmap e68: 6 total, 6 up, 6 in 2024-02-19T19:09:10.670 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:10 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:09:10.670 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:10 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:10 smithi039 ceph-mon[92224]: pgmap v109: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 16/72 objects degraded (22.222%) 2024-02-19T19:09:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:10 smithi039 ceph-mon[92224]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:09:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:10 smithi039 ceph-mon[92224]: osd.3 [v2:172.21.15.86:6800/2382739218,v1:172.21.15.86:6801/2382739218] boot 2024-02-19T19:09:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:10 smithi039 ceph-mon[92224]: osdmap e68: 6 total, 6 up, 6 in 2024-02-19T19:09:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:10 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:09:10.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:10 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 16/72 objects degraded (22.222%), 10 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:11.984 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:11 smithi086 ceph-mon[82675]: osdmap e69: 6 total, 6 up, 6 in 2024-02-19T19:09:11.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:11 smithi039 ceph-mon[92224]: osdmap e69: 6 total, 6 up, 6 in 2024-02-19T19:09:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: pgmap v112: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 16/72 objects degraded (22.222%) 2024-02-19T19:09:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:13.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: pgmap v112: 65 pgs: 10 active+undersized+degraded, 28 active+undersized, 27 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 16/72 objects degraded (22.222%) 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:13.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:14.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:14 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:14.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:14 smithi039 ceph-mon[92224]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-02-19T19:09:14.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:14.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:14 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:14 smithi086 ceph-mon[82675]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-02-19T19:09:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:15.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:15 smithi039 ceph-mon[92224]: pgmap v113: 65 pgs: 8 active+undersized+degraded, 23 active+undersized, 34 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 13/72 objects degraded (18.056%) 2024-02-19T19:09:15.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:15 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:15.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:15 smithi086 ceph-mon[82675]: pgmap v113: 65 pgs: 8 active+undersized+degraded, 23 active+undersized, 34 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 13/72 objects degraded (18.056%) 2024-02-19T19:09:15.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:15 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:16.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:16 smithi039 ceph-mon[92224]: pgmap v114: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:09:16.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:16 smithi039 ceph-mon[92224]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded) 2024-02-19T19:09:16.785 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:16 smithi086 ceph-mon[82675]: pgmap v114: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:09:16.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:16 smithi086 ceph-mon[82675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded) 2024-02-19T19:09:18.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:18 smithi086 ceph-mon[82675]: pgmap v115: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 383 B/s rd, 0 op/s 2024-02-19T19:09:18.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:18 smithi039 ceph-mon[92224]: pgmap v115: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 383 B/s rd, 0 op/s 2024-02-19T19:09:20.888 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:20 smithi039 ceph-mon[92224]: pgmap v116: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 833 B/s rd, 1 op/s 2024-02-19T19:09:21.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:20 smithi086 ceph-mon[82675]: pgmap v116: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 833 B/s rd, 1 op/s 2024-02-19T19:09:22.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:22 smithi039 ceph-mon[92224]: pgmap v117: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 757 B/s rd, 1 op/s 2024-02-19T19:09:23.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:22 smithi086 ceph-mon[82675]: pgmap v117: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 757 B/s rd, 1 op/s 2024-02-19T19:09:24.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:24 smithi039 ceph-mon[92224]: pgmap v118: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2024-02-19T19:09:24.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:24 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:25.035 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:24 smithi086 ceph-mon[82675]: pgmap v118: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2024-02-19T19:09:25.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:24 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:26.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:26 smithi086 ceph-mon[82675]: pgmap v119: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-02-19T19:09:26.832 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:26 smithi039 ceph-mon[92224]: pgmap v119: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-02-19T19:09:27.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:27 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:28.012 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:27 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:28.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:28 smithi086 ceph-mon[82675]: pgmap v120: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:09:28.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:28 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:28.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:28 smithi086 ceph-mon[82675]: Upgrade: osd.4 is safe to restart 2024-02-19T19:09:28.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:28 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:28.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:28 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:28.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:28 smithi039 ceph-mon[92224]: pgmap v120: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:09:28.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:28 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:09:28.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:28 smithi039 ceph-mon[92224]: Upgrade: osd.4 is safe to restart 2024-02-19T19:09:28.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:28 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:28.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:28 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:30.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 -- 172.21.15.39:0/3015145652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 msgr2=0x7f37840ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 --2- 172.21.15.39:0/3015145652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f37840ff160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3770009910 tx=0x7f3770009c20 comp rx=0 tx=0).stop 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 -- 172.21.15.39:0/3015145652 shutdown_connections 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 --2- 172.21.15.39:0/3015145652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f37840ff160 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 --2- 172.21.15.39:0/3015145652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3784102280 0x7f37841026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 -- 172.21.15.39:0/3015145652 >> 172.21.15.39:0/3015145652 conn(0x7f37840fa8c0 msgr2=0x7f37840fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 -- 172.21.15.39:0/3015145652 shutdown_connections 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.074+0000 7f378b142700 1 -- 172.21.15.39:0/3015145652 wait complete. 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f378b142700 1 Processor -- start 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f378b142700 1 -- start start 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f378b142700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f3784196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f378b142700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3784102280 0x7f3784196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f378b142700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37841970b0 con 0x7f37840fed00 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f378b142700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37841971f0 con 0x7f3784102280 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3788ede700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f3784196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3788ede700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f3784196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43330/0 (socket says 172.21.15.39:43330) 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3788ede700 1 -- 172.21.15.39:0/1620766851 learned_addr learned my addr 172.21.15.39:0/1620766851 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3783fff700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3784102280 0x7f3784196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3788ede700 1 -- 172.21.15.39:0/1620766851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3784102280 msgr2=0x7f3784196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3788ede700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3784102280 0x7f3784196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.075+0000 7f3788ede700 1 -- 172.21.15.39:0/1620766851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37700095f0 con 0x7f37840fed00 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.076+0000 7f3788ede700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f3784196550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f377800d900 tx=0x7f377800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.076+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f377800b920 con 0x7f37840fed00 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.076+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378419bca0 con 0x7f37840fed00 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.076+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3778010460 con 0x7f37840fed00 2024-02-19T19:09:30.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.076+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f377800f590 con 0x7f37840fed00 2024-02-19T19:09:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.076+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378419c1a0 con 0x7f37840fed00 2024-02-19T19:09:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.077+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3784109810 con 0x7f37840fed00 2024-02-19T19:09:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.077+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f37780105d0 con 0x7f37840fed00 2024-02-19T19:09:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.078+0000 7f3781ffb700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f377406d8a0 0x7f377406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.078+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f377808d110 con 0x7f37840fed00 2024-02-19T19:09:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.078+0000 7f3783fff700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f377406d8a0 0x7f377406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.078+0000 7f3783fff700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f377406d8a0 0x7f377406fd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3770006010 tx=0x7f377000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.083+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f377805a440 con 0x7f37840fed00 2024-02-19T19:09:30.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.309+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3784106970 con 0x7f377406d8a0 2024-02-19T19:09:30.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.311+0000 7f3781ffb700 1 -- 172.21.15.39:0/1620766851 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f3784106970 con 0x7f377406d8a0 2024-02-19T19:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f377406d8a0 msgr2=0x7f377406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f377406d8a0 0x7f377406fd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3770006010 tx=0x7f377000b540 comp rx=0 tx=0).stop 2024-02-19T19:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 msgr2=0x7f3784196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f3784196550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f377800d900 tx=0x7f377800dc10 comp rx=0 tx=0).stop 2024-02-19T19:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 shutdown_connections 2024-02-19T19:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f377406d8a0 0x7f377406fd60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37840fed00 0x7f3784196550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 --2- 172.21.15.39:0/1620766851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3784102280 0x7f3784196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 >> 172.21.15.39:0/1620766851 conn(0x7f37840fa8c0 msgr2=0x7f3784105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 shutdown_connections 2024-02-19T19:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.313+0000 7f378b142700 1 -- 172.21.15.39:0/1620766851 wait complete. 2024-02-19T19:09:30.345 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:09:30.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 -- 172.21.15.39:0/4225969340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c101130 msgr2=0x7fd36c101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 --2- 172.21.15.39:0/4225969340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c101130 0x7fd36c101550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00b0a0 tx=0x7fd35c00b3b0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 -- 172.21.15.39:0/4225969340 shutdown_connections 2024-02-19T19:09:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 --2- 172.21.15.39:0/4225969340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 --2- 172.21.15.39:0/4225969340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c101130 0x7fd36c101550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 -- 172.21.15.39:0/4225969340 >> 172.21.15.39:0/4225969340 conn(0x7fd36c0fa8c0 msgr2=0x7fd36c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 -- 172.21.15.39:0/4225969340 shutdown_connections 2024-02-19T19:09:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.418+0000 7fd37225a700 1 -- 172.21.15.39:0/4225969340 wait complete. 2024-02-19T19:09:30.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:30 smithi039 ceph-mon[92224]: Upgrade: Updating osd.4 2024-02-19T19:09:30.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:30.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:09:30.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:09:30.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:30 smithi039 ceph-mon[92224]: Deploying daemon osd.4 on smithi086 2024-02-19T19:09:30.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd37225a700 1 Processor -- start 2024-02-19T19:09:30.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd37225a700 1 -- start start 2024-02-19T19:09:30.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd37225a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd37225a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c101130 0x7fd36c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd37225a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c197020 con 0x7fd36c0ffe70 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd37225a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36c197160 con 0x7fd36c101130 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c101130 0x7fd36c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43354/0 (socket says 172.21.15.39:43354) 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36b7fe700 1 -- 172.21.15.39:0/1390166594 learned_addr learned my addr 172.21.15.39:0/1390166594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36b7fe700 1 -- 172.21.15.39:0/1390166594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c101130 msgr2=0x7fd36c196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36b7fe700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c101130 0x7fd36c196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36b7fe700 1 -- 172.21.15.39:0/1390166594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c009d00 con 0x7fd36c0ffe70 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.419+0000 7fd36affd700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c101130 0x7fd36c196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.420+0000 7fd36b7fe700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1964c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00b780 tx=0x7fd35c003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.420+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd35c00e040 con 0x7fd36c0ffe70 2024-02-19T19:09:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.420+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd35c00bdc0 con 0x7fd36c0ffe70 2024-02-19T19:09:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.420+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd35c01d8f0 con 0x7fd36c0ffe70 2024-02-19T19:09:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.420+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36c19bbb0 con 0x7fd36c0ffe70 2024-02-19T19:09:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.420+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36c19c0a0 con 0x7fd36c0ffe70 2024-02-19T19:09:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.421+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd36c108710 con 0x7fd36c0ffe70 2024-02-19T19:09:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.422+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fd35c01dbb0 con 0x7fd36c0ffe70 2024-02-19T19:09:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.422+0000 7fd368ff9700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd35806d8a0 0x7fd35806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.422+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd35c08ef60 con 0x7fd36c0ffe70 2024-02-19T19:09:30.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.422+0000 7fd36affd700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd35806d8a0 0x7fd35806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.423+0000 7fd36affd700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd35806d8a0 0x7fd35806fd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd354011490 tx=0x7fd354009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:30.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.427+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd35c0c2d20 con 0x7fd36c0ffe70 2024-02-19T19:09:30.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:30 smithi086 ceph-mon[82675]: Upgrade: Updating osd.4 2024-02-19T19:09:30.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:30.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:09:30.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:09:30.491 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:30 smithi086 ceph-mon[82675]: Deploying daemon osd.4 on smithi086 2024-02-19T19:09:30.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.653+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd36c0ff470 con 0x7fd35806d8a0 2024-02-19T19:09:30.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.654+0000 7fd368ff9700 1 -- 172.21.15.39:0/1390166594 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fd36c0ff470 con 0x7fd35806d8a0 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd35806d8a0 msgr2=0x7fd35806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd35806d8a0 0x7fd35806fd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd354011490 tx=0x7fd354009260 comp rx=0 tx=0).stop 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 msgr2=0x7fd36c1964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1964c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00b780 tx=0x7fd35c003c30 comp rx=0 tx=0).stop 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 shutdown_connections 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd35806d8a0 0x7fd35806fd60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd36c0ffe70 0x7fd36c1964c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 --2- 172.21.15.39:0/1390166594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd36c101130 0x7fd36c196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 >> 172.21.15.39:0/1390166594 conn(0x7fd36c0fa8c0 msgr2=0x7fd36c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:30.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 shutdown_connections 2024-02-19T19:09:30.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.656+0000 7fd37225a700 1 -- 172.21.15.39:0/1390166594 wait complete. 2024-02-19T19:09:30.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 -- 172.21.15.39:0/3952448939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 msgr2=0x7f4aa4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 --2- 172.21.15.39:0/3952448939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4101200 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009910 tx=0x7f4a94009c20 comp rx=0 tx=0).stop 2024-02-19T19:09:30.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 -- 172.21.15.39:0/3952448939 shutdown_connections 2024-02-19T19:09:30.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 --2- 172.21.15.39:0/3952448939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4101200 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 --2- 172.21.15.39:0/3952448939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa40ffba0 0x7f4aa40fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 -- 172.21.15.39:0/3952448939 >> 172.21.15.39:0/3952448939 conn(0x7f4aa40fb120 msgr2=0x7f4aa40fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 -- 172.21.15.39:0/3952448939 shutdown_connections 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.762+0000 7f4aabb77700 1 -- 172.21.15.39:0/3952448939 wait complete. 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aabb77700 1 Processor -- start 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aabb77700 1 -- start start 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aabb77700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa40ffba0 0x7f4aa41964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aabb77700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aabb77700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4197030 con 0x7f4aa4100da0 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aabb77700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4197170 con 0x7f4aa40ffba0 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9112700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9112700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43358/0 (socket says 172.21.15.39:43358) 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9112700 1 -- 172.21.15.39:0/1981756067 learned_addr learned my addr 172.21.15.39:0/1981756067 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:30.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9913700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa40ffba0 0x7f4aa41964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9112700 1 -- 172.21.15.39:0/1981756067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa40ffba0 msgr2=0x7f4aa41964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9112700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa40ffba0 0x7f4aa41964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.763+0000 7f4aa9112700 1 -- 172.21.15.39:0/1981756067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a940095f0 con 0x7f4aa4100da0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.764+0000 7f4aa9112700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4196a10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4a94006010 tx=0x7f4a9400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.764+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a9401d070 con 0x7f4aa4100da0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.764+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4a9400f460 con 0x7f4aa4100da0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.764+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa419bbc0 con 0x7f4aa4100da0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.764+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a94021640 con 0x7f4aa4100da0 2024-02-19T19:09:30.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.764+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa419c0b0 con 0x7f4aa4100da0 2024-02-19T19:09:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.765+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4108700 con 0x7f4aa4100da0 2024-02-19T19:09:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.765+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f4a9400fab0 con 0x7f4aa4100da0 2024-02-19T19:09:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.766+0000 7f4a9affd700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4a9006d8a0 0x7f4a9006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.766+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4a9408e740 con 0x7f4aa4100da0 2024-02-19T19:09:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.766+0000 7f4aa9913700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4a9006d8a0 0x7f4a9006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:30.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.766+0000 7f4aa9913700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4a9006d8a0 0x7f4a9006fd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009550 tx=0x7f4aa00091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:30.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.770+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a9405ba70 con 0x7f4aa4100da0 2024-02-19T19:09:31.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.984+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4aa4069540 con 0x7f4a9006d8a0 2024-02-19T19:09:31.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.994+0000 7f4a9affd700 1 -- 172.21.15.39:0/1981756067 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f4aa4069540 con 0x7f4a9006d8a0 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (11m) 42s ago 13m 24.8M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (13m) 42s ago 13m 8388k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (12m) 23s ago 12m 7914k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (2m) 42s ago 13m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (2m) 23s ago 12m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (11m) 42s ago 12m 88.3M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (7m) 42s ago 7m 26.4M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (7m) 42s ago 7m 15.4M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (7m) 23s ago 7m 28.9M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (7m) 23s ago 7m 17.4M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (4m) 42s ago 14m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (3m) 23s ago 12m 449M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (3m) 42s ago 14m 46.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (3m) 23s ago 11m 38.3M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:09:31.017 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (13m) 42s ago 13m 15.6M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (12m) 23s ago 12m 16.1M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (108s) 42s ago 11m 51.9M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (71s) 42s ago 10m 51.3M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 546fbde6ffab 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (48s) 42s ago 10m 11.7M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 3d2286828d68 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (26s) 23s ago 9m 11.8M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 37ce649da9a6 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (9m) 23s ago 9m 49.5M 2240M 18.2.1 94eadd990e0a cd7ddf2e286a 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (8m) 23s ago 8m 50.7M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:09:31.018 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (3m) 42s ago 12m 62.3M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4a9006d8a0 msgr2=0x7f4a9006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4a9006d8a0 0x7f4a9006fd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009550 tx=0x7f4aa00091b0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 msgr2=0x7f4aa4196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4196a10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4a94006010 tx=0x7f4a9400bd70 comp rx=0 tx=0).stop 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 shutdown_connections 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4a9006d8a0 0x7f4a9006fd60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa40ffba0 0x7f4aa41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 --2- 172.21.15.39:0/1981756067 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4100da0 0x7f4aa4196a10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 >> 172.21.15.39:0/1981756067 conn(0x7f4aa40fb120 msgr2=0x7f4aa4068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 shutdown_connections 2024-02-19T19:09:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:30.997+0000 7f4aabb77700 1 -- 172.21.15.39:0/1981756067 wait complete. 2024-02-19T19:09:31.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.103+0000 7f7baf22a700 1 -- 172.21.15.39:0/622586053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba80ffc10 msgr2=0x7f7ba8100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.103+0000 7f7baf22a700 1 --2- 172.21.15.39:0/622586053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba80ffc10 0x7f7ba8100030 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009910 tx=0x7f7b9c009c20 comp rx=0 tx=0).stop 2024-02-19T19:09:31.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.103+0000 7f7baf22a700 1 -- 172.21.15.39:0/622586053 shutdown_connections 2024-02-19T19:09:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.103+0000 7f7baf22a700 1 --2- 172.21.15.39:0/622586053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ba8100e10 0x7f7ba8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.103+0000 7f7baf22a700 1 --2- 172.21.15.39:0/622586053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba80ffc10 0x7f7ba8100030 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.103+0000 7f7baf22a700 1 -- 172.21.15.39:0/622586053 >> 172.21.15.39:0/622586053 conn(0x7f7ba80fb190 msgr2=0x7f7ba80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.104+0000 7f7baf22a700 1 -- 172.21.15.39:0/622586053 shutdown_connections 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.104+0000 7f7baf22a700 1 -- 172.21.15.39:0/622586053 wait complete. 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.104+0000 7f7baf22a700 1 Processor -- start 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.104+0000 7f7baf22a700 1 -- start start 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7baf22a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ba80ffc10 0x7f7ba81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7baf22a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 0x7f7ba8198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7baf22a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8199240 con 0x7f7ba8100e10 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7baf22a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8199380 con 0x7f7ba80ffc10 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 0x7f7ba8198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 0x7f7ba8198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43364/0 (socket says 172.21.15.39:43364) 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 -- 172.21.15.39:0/3564797926 learned_addr learned my addr 172.21.15.39:0/3564797926 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7bacfc6700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ba80ffc10 0x7f7ba81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 -- 172.21.15.39:0/3564797926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ba80ffc10 msgr2=0x7f7ba81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ba80ffc10 0x7f7ba81986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 -- 172.21.15.39:0/3564797926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c0095f0 con 0x7f7ba8100e10 2024-02-19T19:09:31.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba7fff700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 0x7f7ba8198c20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400d8d0 tx=0x7f7b9400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:31.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.105+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b94009940 con 0x7f7ba8100e10 2024-02-19T19:09:31.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.106+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7b94010460 con 0x7f7ba8100e10 2024-02-19T19:09:31.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.106+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba819de30 con 0x7f7ba8100e10 2024-02-19T19:09:31.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.106+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9400f5f0 con 0x7f7ba8100e10 2024-02-19T19:09:31.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.106+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba819e350 con 0x7f7ba8100e10 2024-02-19T19:09:31.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.107+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba8108700 con 0x7f7ba8100e10 2024-02-19T19:09:31.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.108+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f7b940105d0 con 0x7f7ba8100e10 2024-02-19T19:09:31.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.108+0000 7f7ba5ffb700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7b9806d8a0 0x7f7b9806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.108+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f7b9408d4a0 con 0x7f7ba8100e10 2024-02-19T19:09:31.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.108+0000 7f7bacfc6700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7b9806d8a0 0x7f7b9806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.109+0000 7f7bacfc6700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7b9806d8a0 0x7f7b9806fd60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0059d0 tx=0x7f7b9c005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:31.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.113+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b9405a7d0 con 0x7f7ba8100e10 2024-02-19T19:09:31.295 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:31 smithi086 ceph-mon[82675]: pgmap v121: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:09:31.295 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:31 smithi086 ceph-mon[82675]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:31.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.403+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7ba81088c0 con 0x7f7ba8100e10 2024-02-19T19:09:31.426 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:31 smithi039 ceph-mon[92224]: pgmap v121: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:09:31.426 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:31 smithi039 ceph-mon[92224]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:31.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.404+0000 7f7ba5ffb700 1 -- 172.21.15.39:0/3564797926 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f7b9405a360 con 0x7f7ba8100e10 2024-02-19T19:09:31.426 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:09:31.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:09:31.428 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:09:31.428 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-02-19T19:09:31.428 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 8 2024-02-19T19:09:31.428 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:09:31.428 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7b9806d8a0 msgr2=0x7f7b9806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7b9806d8a0 0x7f7b9806fd60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0059d0 tx=0x7f7b9c005960 comp rx=0 tx=0).stop 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 msgr2=0x7f7ba8198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 0x7f7ba8198c20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400d8d0 tx=0x7f7b9400dc90 comp rx=0 tx=0).stop 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 shutdown_connections 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7b9806d8a0 0x7f7b9806fd60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ba80ffc10 0x7f7ba81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 --2- 172.21.15.39:0/3564797926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8100e10 0x7f7ba8198c20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 >> 172.21.15.39:0/3564797926 conn(0x7f7ba80fb190 msgr2=0x7f7ba8068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 shutdown_connections 2024-02-19T19:09:31.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.406+0000 7f7baf22a700 1 -- 172.21.15.39:0/3564797926 wait complete. 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 -- 172.21.15.39:0/549129278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 msgr2=0x7f4e3c0ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 --2- 172.21.15.39:0/549129278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c0ff380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009910 tx=0x7f4e38009c20 comp rx=0 tx=0).stop 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 -- 172.21.15.39:0/549129278 shutdown_connections 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 --2- 172.21.15.39:0/549129278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e3c101f40 0x7f4e3c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 --2- 172.21.15.39:0/549129278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c0ff380 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 -- 172.21.15.39:0/549129278 >> 172.21.15.39:0/549129278 conn(0x7f4e3c0fa8c0 msgr2=0x7f4e3c0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:31.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.511+0000 7f4e447ce700 1 -- 172.21.15.39:0/549129278 shutdown_connections 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 -- 172.21.15.39:0/549129278 wait complete. 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 Processor -- start 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 -- start start 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e3c101f40 0x7f4e3c0ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c100430 con 0x7f4e3c0fef60 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e447ce700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c100570 con 0x7f4e3c101f40 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e4256a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e4256a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43390/0 (socket says 172.21.15.39:43390) 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e4256a700 1 -- 172.21.15.39:0/589077643 learned_addr learned my addr 172.21.15.39:0/589077643 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.512+0000 7f4e41d69700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e3c101f40 0x7f4e3c0ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e4256a700 1 -- 172.21.15.39:0/589077643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e3c101f40 msgr2=0x7f4e3c0ffef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e4256a700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e3c101f40 0x7f4e3c0ffef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e4256a700 1 -- 172.21.15.39:0/589077643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e380095f0 con 0x7f4e3c0fef60 2024-02-19T19:09:31.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e4256a700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c1018a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4e38000c00 tx=0x7f4e3800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:31.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e3801d070 con 0x7f4e3c0fef60 2024-02-19T19:09:31.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e38005dc0 con 0x7f4e3c0fef60 2024-02-19T19:09:31.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c1007f0 con 0x7f4e3c0fef60 2024-02-19T19:09:31.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e38022470 con 0x7f4e3c0fef60 2024-02-19T19:09:31.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.513+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c100ce0 con 0x7f4e3c0fef60 2024-02-19T19:09:31.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.514+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c109800 con 0x7f4e3c0fef60 2024-02-19T19:09:31.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.514+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f4e38003810 con 0x7f4e3c0fef60 2024-02-19T19:09:31.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.515+0000 7f4e337fe700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4e2806d8a0 0x7f4e2806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.515+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4e3808e9b0 con 0x7f4e3c0fef60 2024-02-19T19:09:31.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.515+0000 7f4e41d69700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4e2806d8a0 0x7f4e2806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.515+0000 7f4e41d69700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4e2806d8a0 0x7f4e2806fd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009c60 tx=0x7f4e2c009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:31.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.520+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4e3805bce0 con 0x7f4e3c0fef60 2024-02-19T19:09:31.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.770+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4e3c19e930 con 0x7f4e3c0fef60 2024-02-19T19:09:31.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.771+0000 7f4e337fe700 1 -- 172.21.15.39:0/589077643 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f4e3805b870 con 0x7f4e3c0fef60 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:09:31.793 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:09:31.794 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:09:31.795 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:09:31.796 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:09:31.796 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:09:31.796 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:09:31.796 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:09:31.796 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:09:31.796 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4e2806d8a0 msgr2=0x7f4e2806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4e2806d8a0 0x7f4e2806fd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009c60 tx=0x7f4e2c009260 comp rx=0 tx=0).stop 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 msgr2=0x7f4e3c1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c1018a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4e38000c00 tx=0x7f4e3800f690 comp rx=0 tx=0).stop 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 shutdown_connections 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f4e2806d8a0 0x7f4e2806fd60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e3c0fef60 0x7f4e3c1018a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 --2- 172.21.15.39:0/589077643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e3c101f40 0x7f4e3c0ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 >> 172.21.15.39:0/589077643 conn(0x7f4e3c0fa8c0 msgr2=0x7f4e3c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 shutdown_connections 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.773+0000 7f4e447ce700 1 -- 172.21.15.39:0/589077643 wait complete. 2024-02-19T19:09:31.797 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:09:31.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 -- 172.21.15.39:0/1584448402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 msgr2=0x7fda8c101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 --2- 172.21.15.39:0/1584448402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c101220 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda80009910 tx=0x7fda80009c20 comp rx=0 tx=0).stop 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 -- 172.21.15.39:0/1584448402 shutdown_connections 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 --2- 172.21.15.39:0/1584448402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c101220 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 --2- 172.21.15.39:0/1584448402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda8c0ffbc0 0x7fda8c0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 -- 172.21.15.39:0/1584448402 >> 172.21.15.39:0/1584448402 conn(0x7fda8c0fb160 msgr2=0x7fda8c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 -- 172.21.15.39:0/1584448402 shutdown_connections 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.879+0000 7fda913cb700 1 -- 172.21.15.39:0/1584448402 wait complete. 2024-02-19T19:09:31.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda913cb700 1 Processor -- start 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda913cb700 1 -- start start 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda913cb700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda8c0ffbc0 0x7fda8c1956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda913cb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c195c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda913cb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c196220 con 0x7fda8c100dc0 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda913cb700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c196360 con 0x7fda8c0ffbc0 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda8a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c195c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda8a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c195c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43414/0 (socket says 172.21.15.39:43414) 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda8a7fc700 1 -- 172.21.15.39:0/59699107 learned_addr learned my addr 172.21.15.39:0/59699107 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.880+0000 7fda8affd700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda8c0ffbc0 0x7fda8c1956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda8a7fc700 1 -- 172.21.15.39:0/59699107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda8c0ffbc0 msgr2=0x7fda8c1956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda8a7fc700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda8c0ffbc0 0x7fda8c1956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda8a7fc700 1 -- 172.21.15.39:0/59699107 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800095f0 con 0x7fda8c100dc0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda8a7fc700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c195c00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fda80006010 tx=0x7fda8000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda8001f070 con 0x7fda8c100dc0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fda80019610 con 0x7fda8c100dc0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c068e10 con 0x7fda8c100dc0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda8000f690 con 0x7fda8c100dc0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.881+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c069300 con 0x7fda8c100dc0 2024-02-19T19:09:31.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.882+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c069cf0 con 0x7fda8c100dc0 2024-02-19T19:09:31.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.883+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fda80019c60 con 0x7fda8c100dc0 2024-02-19T19:09:31.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.883+0000 7fda73fff700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fda7406d8a0 0x7fda7406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:31.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.883+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fda80090990 con 0x7fda8c100dc0 2024-02-19T19:09:31.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.883+0000 7fda8affd700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fda7406d8a0 0x7fda7406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:31.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.883+0000 7fda8affd700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fda7406d8a0 0x7fda7406fd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda8c100c20 tx=0x7fda7c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:31.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:31.888+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda8005dc60 con 0x7fda8c100dc0 2024-02-19T19:09:32.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.112+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda8c18f1b0 con 0x7fda7406d8a0 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.114+0000 7fda73fff700 1 -- 172.21.15.39:0/59699107 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fda8c18f1b0 con 0x7fda7406d8a0 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:09:32.136 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:09:32.137 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:09:32.137 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "10/23 daemons upgraded", 2024-02-19T19:09:32.137 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:09:32.137 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:09:32.137 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:09:32.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fda7406d8a0 msgr2=0x7fda7406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:32.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fda7406d8a0 0x7fda7406fd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda8c100c20 tx=0x7fda7c00b410 comp rx=0 tx=0).stop 2024-02-19T19:09:32.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 msgr2=0x7fda8c195c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:32.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c195c00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fda80006010 tx=0x7fda8000bd70 comp rx=0 tx=0).stop 2024-02-19T19:09:32.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 shutdown_connections 2024-02-19T19:09:32.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fda7406d8a0 0x7fda7406fd60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda8c0ffbc0 0x7fda8c1956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 --2- 172.21.15.39:0/59699107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda8c100dc0 0x7fda8c195c00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 >> 172.21.15.39:0/59699107 conn(0x7fda8c0fb160 msgr2=0x7fda8c104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:32.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 shutdown_connections 2024-02-19T19:09:32.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.116+0000 7fda913cb700 1 -- 172.21.15.39:0/59699107 wait complete. 2024-02-19T19:09:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 -- 172.21.15.39:0/212365832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec0ffc10 msgr2=0x7fafec100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 --2- 172.21.15.39:0/212365832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec0ffc10 0x7fafec100030 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009910 tx=0x7fafdc009c20 comp rx=0 tx=0).stop 2024-02-19T19:09:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 -- 172.21.15.39:0/212365832 shutdown_connections 2024-02-19T19:09:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 --2- 172.21.15.39:0/212365832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafec100e10 0x7fafec101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 --2- 172.21.15.39:0/212365832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec0ffc10 0x7fafec100030 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 -- 172.21.15.39:0/212365832 >> 172.21.15.39:0/212365832 conn(0x7fafec0fb190 msgr2=0x7fafec0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:32.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 -- 172.21.15.39:0/212365832 shutdown_connections 2024-02-19T19:09:32.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 -- 172.21.15.39:0/212365832 wait complete. 2024-02-19T19:09:32.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.222+0000 7faff175d700 1 Processor -- start 2024-02-19T19:09:32.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7faff175d700 1 -- start start 2024-02-19T19:09:32.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7faff175d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafec0ffc10 0x7fafec1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:32.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7faff175d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 0x7fafec196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:32.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7faff175d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec197030 con 0x7fafec100e10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7faff175d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec197170 con 0x7fafec0ffc10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 0x7fafec196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 0x7fafec196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43440/0 (socket says 172.21.15.39:43440) 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 -- 172.21.15.39:0/2386130070 learned_addr learned my addr 172.21.15.39:0/2386130070 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafeaffd700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafec0ffc10 0x7fafec1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 -- 172.21.15.39:0/2386130070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafec0ffc10 msgr2=0x7fafec1964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafec0ffc10 0x7fafec1964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 -- 172.21.15.39:0/2386130070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0095f0 con 0x7fafec100e10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.223+0000 7fafea7fc700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 0x7fafec196a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fafd400e920 tx=0x7fafd400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.224+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd400c9d0 con 0x7fafec100e10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.224+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec19bc20 con 0x7fafec100e10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.224+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fafd4010430 con 0x7fafec100e10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.224+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd4018650 con 0x7fafec100e10 2024-02-19T19:09:32.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.224+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec19c170 con 0x7fafec100e10 2024-02-19T19:09:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.225+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec108700 con 0x7fafec100e10 2024-02-19T19:09:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.225+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fafd400cb30 con 0x7fafec100e10 2024-02-19T19:09:32.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.225+0000 7fafe3fff700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fafd806d8a0 0x7fafd806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:09:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.226+0000 7fafeaffd700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fafd806d8a0 0x7fafd806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:09:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.226+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fafd4014070 con 0x7fafec100e10 2024-02-19T19:09:32.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.226+0000 7fafeaffd700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fafd806d8a0 0x7fafd806fd60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b6f0 tx=0x7fafdc005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:09:32.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.230+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fafd4052d20 con 0x7fafec100e10 2024-02-19T19:09:32.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:32 smithi039 ceph-mon[92224]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:32.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:32 smithi039 ceph-mon[92224]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:32.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:32 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3564797926' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:32.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:32 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/589077643' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:09:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-mon[82675]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-mon[82675]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3564797926' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:32.533 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/589077643' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:09:32.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.529+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fafec19c780 con 0x7fafec100e10 2024-02-19T19:09:32.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.529+0000 7fafe3fff700 1 -- 172.21.15.39:0/2386130070 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fafd405ab10 con 0x7fafec100e10 2024-02-19T19:09:32.551 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:09:32.552 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:09:32.552 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fafd806d8a0 msgr2=0x7fafd806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fafd806d8a0 0x7fafd806fd60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b6f0 tx=0x7fafdc005940 comp rx=0 tx=0).stop 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 msgr2=0x7fafec196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 0x7fafec196a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fafd400e920 tx=0x7fafd400ec30 comp rx=0 tx=0).stop 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 shutdown_connections 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fafd806d8a0 0x7fafd806fd60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafec0ffc10 0x7fafec1964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 --2- 172.21.15.39:0/2386130070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafec100e10 0x7fafec196a10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:09:32.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 >> 172.21.15.39:0/2386130070 conn(0x7fafec0fb190 msgr2=0x7fafec068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:09:32.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 shutdown_connections 2024-02-19T19:09:32.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:09:32.532+0000 7faff175d700 1 -- 172.21.15.39:0/2386130070 wait complete. 2024-02-19T19:09:32.786 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:32 smithi086 systemd[1]: Stopping Ceph osd.4 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:09:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:33 smithi086 ceph-mon[82675]: pgmap v122: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:09:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:33 smithi086 ceph-mon[82675]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:33 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/2386130070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:09:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:33 smithi086 ceph-mon[82675]: osd.4 marked itself down and dead 2024-02-19T19:09:33.286 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[50055]: 2024-02-19T19:09:32.959+0000 7f5ea5499700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:09:33.286 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[50055]: 2024-02-19T19:09:32.959+0000 7f5ea5499700 -1 osd.4 69 *** Got signal Terminated *** 2024-02-19T19:09:33.286 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:32 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[50055]: 2024-02-19T19:09:32.959+0000 7f5ea5499700 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:09:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:33 smithi039 ceph-mon[92224]: pgmap v122: 65 pgs: 65 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:09:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:33 smithi039 ceph-mon[92224]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:09:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:33 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/2386130070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:09:33.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:33 smithi039 ceph-mon[92224]: osd.4 marked itself down and dead 2024-02-19T19:09:33.939 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:33 smithi086 podman[97513]: 2024-02-19 19:09:33.6122427 +0000 UTC m=+0.760282201 container died cd7ddf2e286a6ec585df879a697bd6b0e0f45bebf2b144a4e3769347d4fef199 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:09:34.213 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:34 smithi086 podman[97513]: 2024-02-19 19:09:34.093286098 +0000 UTC m=+1.241325637 container cleanup cd7ddf2e286a6ec585df879a697bd6b0e0f45bebf2b144a4e3769347d4fef199 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-19T19:09:34.213 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:34 smithi086 bash[97513]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4 2024-02-19T19:09:34.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:34 smithi039 ceph-mon[92224]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:09:34.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:34 smithi039 ceph-mon[92224]: osdmap e70: 6 total, 5 up, 6 in 2024-02-19T19:09:34.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:34 smithi039 ceph-mon[92224]: osdmap e71: 6 total, 5 up, 6 in 2024-02-19T19:09:34.536 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:34 smithi086 podman[97531]: 2024-02-19 19:09:34.378480238 +0000 UTC m=+0.764726550 container remove cd7ddf2e286a6ec585df879a697bd6b0e0f45bebf2b144a4e3769347d4fef199 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-19T19:09:34.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:34 smithi086 ceph-mon[82675]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:09:34.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:34 smithi086 ceph-mon[82675]: osdmap e70: 6 total, 5 up, 6 in 2024-02-19T19:09:34.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:34 smithi086 ceph-mon[82675]: osdmap e71: 6 total, 5 up, 6 in 2024-02-19T19:09:35.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:34 smithi086 podman[97622]: 2024-02-19 19:09:34.611836742 +0000 UTC m=+0.017960664 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:09:35.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:34 smithi086 podman[97622]: 2024-02-19 19:09:34.738024978 +0000 UTC m=+0.144148894 container create a59be91141279e45ac9cb62f53deff599dde47168aa2ea8e6ca5dec6d2340593 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-19T19:09:35.355 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:35 smithi086 ceph-mon[82675]: pgmap v124: 65 pgs: 6 stale+active+clean, 59 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:09:35.356 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 podman[97622]: 2024-02-19 19:09:35.041486318 +0000 UTC m=+0.447610232 container init a59be91141279e45ac9cb62f53deff599dde47168aa2ea8e6ca5dec6d2340593 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-19T19:09:35.356 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 podman[97622]: 2024-02-19 19:09:35.044756019 +0000 UTC m=+0.450879936 container start a59be91141279e45ac9cb62f53deff599dde47168aa2ea8e6ca5dec6d2340593 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-19T19:09:35.356 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 podman[97622]: 2024-02-19 19:09:35.140633857 +0000 UTC m=+0.546757772 container attach a59be91141279e45ac9cb62f53deff599dde47168aa2ea8e6ca5dec6d2340593 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:09:35.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:35 smithi039 ceph-mon[92224]: pgmap v124: 65 pgs: 6 stale+active+clean, 59 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:09:35.761 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 podman[97622]: 2024-02-19 19:09:35.33528254 +0000 UTC m=+0.741406460 container died a59be91141279e45ac9cb62f53deff599dde47168aa2ea8e6ca5dec6d2340593 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:09:36.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 podman[97647]: 2024-02-19 19:09:35.894630761 +0000 UTC m=+0.550067608 container remove a59be91141279e45ac9cb62f53deff599dde47168aa2ea8e6ca5dec6d2340593 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:09:36.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-19T19:09:36.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 systemd[1]: Stopped Ceph osd.4 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:09:36.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:35 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.4.service: Consumed 5.007s CPU time. 2024-02-19T19:09:36.399 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:36 smithi039 ceph-mon[92224]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:36.490 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:36 smithi086 ceph-mon[82675]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:36.491 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:36 smithi086 systemd[1]: Starting Ceph osd.4 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:09:36.491 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:36 smithi086 podman[97748]: 2024-02-19 19:09:36.231588381 +0000 UTC m=+0.018342054 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:09:36.491 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:36 smithi086 podman[97748]: 2024-02-19 19:09:36.346381171 +0000 UTC m=+0.133134834 container create a6d4859be19a3c6726fc6f6e04b19eb06c64ceeff7bdc11f1411018d1125e799 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-19T19:09:36.786 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:36 smithi086 podman[97748]: 2024-02-19 19:09:36.607170071 +0000 UTC m=+0.393923735 container init a6d4859be19a3c6726fc6f6e04b19eb06c64ceeff7bdc11f1411018d1125e799 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-19T19:09:36.786 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:36 smithi086 podman[97748]: 2024-02-19 19:09:36.610977562 +0000 UTC m=+0.397731244 container start a6d4859be19a3c6726fc6f6e04b19eb06c64ceeff7bdc11f1411018d1125e799 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:09:36.786 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:36 smithi086 podman[97748]: 2024-02-19 19:09:36.707371122 +0000 UTC m=+0.494124798 container attach a6d4859be19a3c6726fc6f6e04b19eb06c64ceeff7bdc11f1411018d1125e799 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=) 2024-02-19T19:09:37.439 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-mon[82675]: pgmap v126: 65 pgs: 2 active+undersized+degraded, 11 peering, 3 stale+active+clean, 11 active+undersized, 38 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 bash[97748]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-19T19:09:37.440 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:09:37.441 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:09:37.441 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-02-19T19:09:37.441 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:09:37.441 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate[97762]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-19T19:09:37.441 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 podman[97748]: 2024-02-19 19:09:37.073244127 +0000 UTC m=+0.859997793 container died a6d4859be19a3c6726fc6f6e04b19eb06c64ceeff7bdc11f1411018d1125e799 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5) 2024-02-19T19:09:37.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:37 smithi039 ceph-mon[92224]: pgmap v126: 65 pgs: 2 active+undersized+degraded, 11 peering, 3 stale+active+clean, 11 active+undersized, 38 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-02-19T19:09:38.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 podman[97898]: 2024-02-19 19:09:37.728112417 +0000 UTC m=+0.649798758 container remove a6d4859be19a3c6726fc6f6e04b19eb06c64ceeff7bdc11f1411018d1125e799 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-19T19:09:38.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 podman[97962]: 2024-02-19 19:09:37.84375163 +0000 UTC m=+0.017912640 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:09:38.036 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:37 smithi086 podman[97962]: 2024-02-19 19:09:37.958886245 +0000 UTC m=+0.133047242 container create f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-19T19:09:38.536 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:38 smithi086 podman[97962]: 2024-02-19 19:09:38.356857444 +0000 UTC m=+0.531018443 container init f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:09:38.536 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:38 smithi086 podman[97962]: 2024-02-19 19:09:38.36008288 +0000 UTC m=+0.534243881 container start f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-19T19:09:38.536 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:38 smithi086 ceph-osd[97981]: -- 172.21.15.86:0/242635881 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567363c31e0 con 0x556736395000 2024-02-19T19:09:38.536 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:38 smithi086 bash[97962]: f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee 2024-02-19T19:09:38.536 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:38 smithi086 systemd[1]: Started Ceph osd.4 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:09:39.270 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:38 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[97977]: 2024-02-19T19:09:38.905+0000 7fdafbd6b7c0 -1 Falling back to public interface 2024-02-19T19:09:39.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:39 smithi086 ceph-mon[82675]: pgmap v127: 65 pgs: 5 active+undersized+degraded, 11 peering, 18 active+undersized, 31 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-02-19T19:09:39.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:39.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:39.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:39 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:39.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:39 smithi039 ceph-mon[92224]: pgmap v127: 65 pgs: 5 active+undersized+degraded, 11 peering, 18 active+undersized, 31 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 7/72 objects degraded (9.722%) 2024-02-19T19:09:39.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:39.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:39.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:39 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:41.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:41 smithi039 ceph-mon[92224]: pgmap v128: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-02-19T19:09:41.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:41 smithi086 ceph-mon[82675]: pgmap v128: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-02-19T19:09:42.786 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[97977]: 2024-02-19T19:09:42.381+0000 7fdafbd6b7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-19T19:09:42.786 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[97977]: 2024-02-19T19:09:42.409+0000 7fdafbd6b7c0 -1 osd.4 69 log_to_monitors true 2024-02-19T19:09:42.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-mon[82675]: pgmap v129: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-02-19T19:09:42.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:42.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:42.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-mon[82675]: from='osd.4 [v2:172.21.15.86:6808/560609481,v1:172.21.15.86:6809/560609481]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:09:42.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:42 smithi086 ceph-mon[82675]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:09:42.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:42 smithi039 ceph-mon[92224]: pgmap v129: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-02-19T19:09:42.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:42.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:42.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:42 smithi039 ceph-mon[92224]: from='osd.4 [v2:172.21.15.86:6808/560609481,v1:172.21.15.86:6809/560609481]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:09:42.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:42 smithi039 ceph-mon[92224]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:09:43.610 INFO:journalctl@ceph.osd.4.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-4[97977]: 2024-02-19T19:09:43.289+0000 7fdaed1c3700 -1 osd.4 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:09:43.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:43.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:43.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T19:09:43.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='osd.4 [v2:172.21.15.86:6808/560609481,v1:172.21.15.86:6809/560609481]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:43.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: osdmap e72: 6 total, 5 up, 6 in 2024-02-19T19:09:43.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:43.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:43.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='osd.4 [v2:172.21.15.86:6808/560609481,v1:172.21.15.86:6809/560609481]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: osdmap e72: 6 total, 5 up, 6 in 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:44.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:44.960 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:44 smithi086 ceph-mon[82675]: pgmap v131: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 13/72 objects degraded (18.056%) 2024-02-19T19:09:44.960 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:44 smithi086 ceph-mon[82675]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:09:44.960 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:44 smithi086 ceph-mon[82675]: osd.4 [v2:172.21.15.86:6808/560609481,v1:172.21.15.86:6809/560609481] boot 2024-02-19T19:09:44.960 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:44 smithi086 ceph-mon[82675]: osdmap e73: 6 total, 6 up, 6 in 2024-02-19T19:09:44.960 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:09:44.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:44 smithi039 ceph-mon[92224]: pgmap v131: 65 pgs: 8 active+undersized+degraded, 26 active+undersized, 31 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 13/72 objects degraded (18.056%) 2024-02-19T19:09:44.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:44 smithi039 ceph-mon[92224]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:09:44.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:44 smithi039 ceph-mon[92224]: osd.4 [v2:172.21.15.86:6808/560609481,v1:172.21.15.86:6809/560609481] boot 2024-02-19T19:09:44.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:44 smithi039 ceph-mon[92224]: osdmap e73: 6 total, 6 up, 6 in 2024-02-19T19:09:44.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:09:45.980 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:45 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:45.980 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:45 smithi086 ceph-mon[82675]: osdmap e74: 6 total, 6 up, 6 in 2024-02-19T19:09:45.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:45 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:45.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:45 smithi039 ceph-mon[92224]: osdmap e74: 6 total, 6 up, 6 in 2024-02-19T19:09:47.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: pgmap v134: 65 pgs: 4 peering, 8 active+undersized+degraded, 22 active+undersized, 31 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 13/72 objects degraded (18.056%) 2024-02-19T19:09:47.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:47 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: pgmap v134: 65 pgs: 4 peering, 8 active+undersized+degraded, 22 active+undersized, 31 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 13/72 objects degraded (18.056%) 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:47.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:09:47.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:47 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:09:48.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:48 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:09:48.752 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:48 smithi039 ceph-mon[92224]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-19T19:09:48.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:48 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:09:48.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:48 smithi086 ceph-mon[82675]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-19T19:09:49.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:49 smithi039 ceph-mon[92224]: pgmap v135: 65 pgs: 4 peering, 6 active+undersized+degraded, 17 active+undersized, 38 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 9/72 objects degraded (12.500%) 2024-02-19T19:09:49.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:49 smithi086 ceph-mon[82675]: pgmap v135: 65 pgs: 4 peering, 6 active+undersized+degraded, 17 active+undersized, 38 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s; 9/72 objects degraded (12.500%) 2024-02-19T19:09:50.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:50 smithi039 ceph-mon[92224]: pgmap v136: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:09:50.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:50 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:50.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:50 smithi086 ceph-mon[82675]: pgmap v136: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:09:50.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:50 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded (PG_DEGRADED) 2024-02-19T19:09:51.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:51 smithi039 ceph-mon[92224]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded) 2024-02-19T19:09:51.753 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:51 smithi086 ceph-mon[82675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/72 objects degraded (12.500%), 6 pgs degraded) 2024-02-19T19:09:52.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:52 smithi039 ceph-mon[92224]: pgmap v137: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-02-19T19:09:52.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:52 smithi086 ceph-mon[82675]: pgmap v137: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-02-19T19:09:54.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:54 smithi039 ceph-mon[92224]: pgmap v138: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 445 B/s rd, 0 op/s 2024-02-19T19:09:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:54 smithi086 ceph-mon[82675]: pgmap v138: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 445 B/s rd, 0 op/s 2024-02-19T19:09:56.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:56 smithi039 ceph-mon[92224]: pgmap v139: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 904 B/s rd, 1 op/s 2024-02-19T19:09:56.809 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:56 smithi086 ceph-mon[82675]: pgmap v139: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 904 B/s rd, 1 op/s 2024-02-19T19:09:58.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:58 smithi039 ceph-mon[92224]: pgmap v140: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T19:09:58.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:58.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:59.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:58 smithi086 ceph-mon[82675]: pgmap v140: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T19:09:59.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:09:59.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:09:59.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:09:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:00.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:09:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:00 smithi039 ceph-mon[92224]: pgmap v141: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:10:00.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:00 smithi039 ceph-mon[92224]: overall HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:10:01.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:00 smithi086 ceph-mon[82675]: pgmap v141: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:10:01.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:00 smithi086 ceph-mon[82675]: overall HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:10:01.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:01 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:10:01.864 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:01 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 -- 172.21.15.39:0/2305558154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 msgr2=0x7f8514102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2305558154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514102310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8508009910 tx=0x7f8508009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 -- 172.21.15.39:0/2305558154 shutdown_connections 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2305558154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514102310 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2305558154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 0x7f85141010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 -- 172.21.15.39:0/2305558154 >> 172.21.15.39:0/2305558154 conn(0x7f85140fc230 msgr2=0x7f85140fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 -- 172.21.15.39:0/2305558154 shutdown_connections 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.651+0000 7f851b6c6700 1 -- 172.21.15.39:0/2305558154 wait complete. 2024-02-19T19:10:02.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f851b6c6700 1 Processor -- start 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f851b6c6700 1 -- start start 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f851b6c6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 0x7f85141964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f851b6c6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f851b6c6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514197050 con 0x7f8514101eb0 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f851b6c6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8514197190 con 0x7f8514100cb0 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f8518c61700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f8518c61700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54686/0 (socket says 172.21.15.39:54686) 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f8518c61700 1 -- 172.21.15.39:0/2751668276 learned_addr learned my addr 172.21.15.39:0/2751668276 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.652+0000 7f8519462700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 0x7f85141964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f8518c61700 1 -- 172.21.15.39:0/2751668276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 msgr2=0x7f85141964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f8518c61700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 0x7f85141964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:02.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f8518c61700 1 -- 172.21.15.39:0/2751668276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85080095f0 con 0x7f8514101eb0 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f8519462700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 0x7f85141964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f8518c61700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514196a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8508000c00 tx=0x7f850800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f850801d070 con 0x7f8514101eb0 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8508005dc0 con 0x7f8514101eb0 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85080226a0 con 0x7f8514101eb0 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851419bbe0 con 0x7f8514101eb0 2024-02-19T19:10:02.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.653+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851419c0d0 con 0x7f8514101eb0 2024-02-19T19:10:02.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.654+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8514109810 con 0x7f8514101eb0 2024-02-19T19:10:02.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.655+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f85080038f0 con 0x7f8514101eb0 2024-02-19T19:10:02.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.655+0000 7f85067fc700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f850006d8a0 0x7f850006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:02.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.655+0000 7f8519462700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f850006d8a0 0x7f850006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:02.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.655+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f850808e8d0 con 0x7f8514101eb0 2024-02-19T19:10:02.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.656+0000 7f8519462700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f850006d8a0 0x7f850006fd60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8514101d10 tx=0x7f8510009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.660+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f850805bc80 con 0x7f8514101eb0 2024-02-19T19:10:02.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:02 smithi086 ceph-mon[82675]: pgmap v142: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:10:02.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:02 smithi086 ceph-mon[82675]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:10:02.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:02 smithi086 ceph-mon[82675]: Upgrade: osd.5 is safe to restart 2024-02-19T19:10:02.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.886+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8514106800 con 0x7f850006d8a0 2024-02-19T19:10:02.909 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:02 smithi039 ceph-mon[92224]: pgmap v142: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:10:02.909 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:02 smithi039 ceph-mon[92224]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:10:02.909 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:02 smithi039 ceph-mon[92224]: Upgrade: osd.5 is safe to restart 2024-02-19T19:10:02.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.887+0000 7f85067fc700 1 -- 172.21.15.39:0/2751668276 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f8514106800 con 0x7f850006d8a0 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f850006d8a0 msgr2=0x7f850006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f850006d8a0 0x7f850006fd60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8514101d10 tx=0x7f8510009260 comp rx=0 tx=0).stop 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 msgr2=0x7f8514196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514196a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8508000c00 tx=0x7f850800f690 comp rx=0 tx=0).stop 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 shutdown_connections 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f850006d8a0 0x7f850006fd60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.889+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8514100cb0 0x7f85141964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:02.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.890+0000 7f851b6c6700 1 --2- 172.21.15.39:0/2751668276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514101eb0 0x7f8514196a30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.890+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 >> 172.21.15.39:0/2751668276 conn(0x7f85140fc230 msgr2=0x7f85141050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.890+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 shutdown_connections 2024-02-19T19:10:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.890+0000 7f851b6c6700 1 -- 172.21.15.39:0/2751668276 wait complete. 2024-02-19T19:10:02.922 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:10:03.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 -- 172.21.15.39:0/893381100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10104080 msgr2=0x7f1d10106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 --2- 172.21.15.39:0/893381100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10104080 0x7f1d10106470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009910 tx=0x7f1d04009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:03.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 -- 172.21.15.39:0/893381100 shutdown_connections 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 --2- 172.21.15.39:0/893381100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10104080 0x7f1d10106470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 --2- 172.21.15.39:0/893381100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10101750 0x7f1d10103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 -- 172.21.15.39:0/893381100 >> 172.21.15.39:0/893381100 conn(0x7f1d100fb160 msgr2=0x7f1d100fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 -- 172.21.15.39:0/893381100 shutdown_connections 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 -- 172.21.15.39:0/893381100 wait complete. 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 Processor -- start 2024-02-19T19:10:03.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.997+0000 7f1d1657d700 1 -- start start 2024-02-19T19:10:03.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d1657d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 0x7f1d10198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d1657d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10104080 0x7f1d10198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d1657d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101994e0 con 0x7f1d10101750 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d1657d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10199620 con 0x7f1d10104080 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 0x7f1d10198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 0x7f1d10198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54716/0 (socket says 172.21.15.39:54716) 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 -- 172.21.15.39:0/1201246325 learned_addr learned my addr 172.21.15.39:0/1201246325 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0f7fe700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10104080 0x7f1d10198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 -- 172.21.15.39:0/1201246325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10104080 msgr2=0x7f1d10198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10104080 0x7f1d10198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 -- 172.21.15.39:0/1201246325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040095f0 con 0x7f1d10101750 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0f7fe700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10104080 0x7f1d10198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0ffff700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 0x7f1d10198980 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000d8d0 tx=0x7f1d0000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d00009940 con 0x7f1d10101750 2024-02-19T19:10:03.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1d00010460 con 0x7f1d10101750 2024-02-19T19:10:03.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1019e0d0 con 0x7f1d10101750 2024-02-19T19:10:03.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d0000f5f0 con 0x7f1d10101750 2024-02-19T19:10:03.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.998+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1019e620 con 0x7f1d10101750 2024-02-19T19:10:03.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:02.999+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d10068af0 con 0x7f1d10101750 2024-02-19T19:10:03.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.000+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f1d000105d0 con 0x7f1d10101750 2024-02-19T19:10:03.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.000+0000 7f1d0d7fa700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1cf806d8a0 0x7f1cf806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.000+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1d0008d3e0 con 0x7f1d10101750 2024-02-19T19:10:03.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.000+0000 7f1d0f7fe700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1cf806d8a0 0x7f1cf806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.001+0000 7f1d0f7fe700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1cf806d8a0 0x7f1cf806fd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1d04006010 tx=0x7f1d0400b580 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:03.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.005+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1d0005a790 con 0x7f1d10101750 2024-02-19T19:10:03.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.231+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d10100e10 con 0x7f1cf806d8a0 2024-02-19T19:10:03.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.232+0000 7f1d0d7fa700 1 -- 172.21.15.39:0/1201246325 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f1d10100e10 con 0x7f1cf806d8a0 2024-02-19T19:10:03.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1cf806d8a0 msgr2=0x7f1cf806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1cf806d8a0 0x7f1cf806fd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1d04006010 tx=0x7f1d0400b580 comp rx=0 tx=0).stop 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 msgr2=0x7f1d10198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 0x7f1d10198980 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000d8d0 tx=0x7f1d0000dc90 comp rx=0 tx=0).stop 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 shutdown_connections 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1cf806d8a0 0x7f1cf806fd60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d10101750 0x7f1d10198980 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 --2- 172.21.15.39:0/1201246325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d10104080 0x7f1d10198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 >> 172.21.15.39:0/1201246325 conn(0x7f1d100fb160 msgr2=0x7f1d10100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 shutdown_connections 2024-02-19T19:10:03.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.234+0000 7f1d1657d700 1 -- 172.21.15.39:0/1201246325 wait complete. 2024-02-19T19:10:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.342+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/1149791337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481040d0 msgr2=0x7f9c481064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.342+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/1149791337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481040d0 0x7f9c481064c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009910 tx=0x7f9c3c009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.342+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/1149791337 shutdown_connections 2024-02-19T19:10:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.342+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/1149791337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481040d0 0x7f9c481064c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.342+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/1149791337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c481017a0 0x7f9c48103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.342+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/1149791337 >> 172.21.15.39:0/1149791337 conn(0x7f9c480fb190 msgr2=0x7f9c480fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/1149791337 shutdown_connections 2024-02-19T19:10:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/1149791337 wait complete. 2024-02-19T19:10:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 Processor -- start 2024-02-19T19:10:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 -- start start 2024-02-19T19:10:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 0x7f9c48196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c481040d0 0x7f9c48196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48196fb0 con 0x7f9c481017a0 2024-02-19T19:10:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.343+0000 7f9c4d9ae700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c481970f0 con 0x7f9c481040d0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 0x7f9c48196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 0x7f9c48196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54734/0 (socket says 172.21.15.39:54734) 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 -- 172.21.15.39:0/3973908037 learned_addr learned my addr 172.21.15.39:0/3973908037 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c467fc700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c481040d0 0x7f9c48196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 -- 172.21.15.39:0/3973908037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c481040d0 msgr2=0x7f9c48196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c481040d0 0x7f9c48196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 -- 172.21.15.39:0/3973908037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c0095f0 con 0x7f9c481017a0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c46ffd700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 0x7f9c48196450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800e920 tx=0x7f9c3800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c3800c9d0 con 0x7f9c481017a0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9c38010430 con 0x7f9c481017a0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4819bba0 con 0x7f9c481017a0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c38018650 con 0x7f9c481017a0 2024-02-19T19:10:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.344+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4819c0f0 con 0x7f9c481017a0 2024-02-19T19:10:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.346+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c48190610 con 0x7f9c481017a0 2024-02-19T19:10:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.346+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f9c380105a0 con 0x7f9c481017a0 2024-02-19T19:10:03.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.346+0000 7f9c4c9ac700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9c3006d8a0 0x7f9c3006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.346+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f9c38014070 con 0x7f9c481017a0 2024-02-19T19:10:03.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.346+0000 7f9c467fc700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9c3006d8a0 0x7f9c3006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.347+0000 7f9c467fc700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9c3006d8a0 0x7f9c3006fd60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009fd0 tx=0x7f9c3c005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:03.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.353+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c38052d20 con 0x7f9c481017a0 2024-02-19T19:10:03.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.571+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c480611d0 con 0x7f9c3006d8a0 2024-02-19T19:10:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.582+0000 7f9c4c9ac700 1 -- 172.21.15.39:0/3973908037 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f9c480611d0 con 0x7f9c3006d8a0 2024-02-19T19:10:03.605 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:10:03.605 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (12m) 75s ago 13m 24.8M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:10:03.605 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (14m) 75s ago 14m 8388k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:10:03.606 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (12m) 22s ago 12m 8150k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:10:03.606 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (2m) 75s ago 13m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:10:03.606 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (2m) 22s ago 12m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:10:03.606 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (12m) 75s ago 13m 88.3M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:10:03.606 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (7m) 75s ago 7m 26.4M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:10:03.607 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (8m) 75s ago 8m 15.4M - 18.2.1 94eadd990e0a 7fce10faef2b 2024-02-19T19:10:03.607 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (7m) 22s ago 7m 29.2M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:10:03.607 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (7m) 22s ago 7m 17.7M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:10:03.607 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (4m) 75s ago 15m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:10:03.607 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (4m) 22s ago 12m 450M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:10:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (3m) 75s ago 15m 46.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:10:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (3m) 22s ago 12m 36.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:10:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (13m) 75s ago 13m 15.6M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:10:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (12m) 22s ago 12m 16.1M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:10:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (2m) 75s ago 11m 51.9M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:10:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (103s) 75s ago 11m 51.3M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 546fbde6ffab 2024-02-19T19:10:03.609 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (81s) 75s ago 10m 11.7M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 3d2286828d68 2024-02-19T19:10:03.609 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (59s) 22s ago 10m 44.6M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 37ce649da9a6 2024-02-19T19:10:03.609 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (25s) 22s ago 9m 14.8M 2240M 19.0.0-1450-gf78a58c0 f081f847070a f6b44d6c5641 2024-02-19T19:10:03.609 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (9m) 22s ago 9m 52.0M 2240M 18.2.1 94eadd990e0a 439153b12a3d 2024-02-19T19:10:03.609 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (4m) 75s ago 13m 62.3M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:10:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.584+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9c3006d8a0 msgr2=0x7f9c3006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.584+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9c3006d8a0 0x7f9c3006fd60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009fd0 tx=0x7f9c3c005fd0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 msgr2=0x7f9c48196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 0x7f9c48196450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800e920 tx=0x7f9c3800ec30 comp rx=0 tx=0).stop 2024-02-19T19:10:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 shutdown_connections 2024-02-19T19:10:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9c3006d8a0 0x7f9c3006fd60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c481017a0 0x7f9c48196450 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 --2- 172.21.15.39:0/3973908037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c481040d0 0x7f9c48196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 >> 172.21.15.39:0/3973908037 conn(0x7f9c480fb190 msgr2=0x7f9c480fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 shutdown_connections 2024-02-19T19:10:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.585+0000 7f9c4d9ae700 1 -- 172.21.15.39:0/3973908037 wait complete. 2024-02-19T19:10:03.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.688+0000 7f98f761f700 1 -- 172.21.15.39:0/3479572447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f00fed00 msgr2=0x7f98f00ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.688+0000 7f98f761f700 1 --2- 172.21.15.39:0/3479572447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f00fed00 0x7f98f00ff160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009910 tx=0x7f98e0009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:03.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- 172.21.15.39:0/3479572447 shutdown_connections 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 --2- 172.21.15.39:0/3479572447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f00fed00 0x7f98f00ff160 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 --2- 172.21.15.39:0/3479572447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98f0102280 0x7f98f01026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- 172.21.15.39:0/3479572447 >> 172.21.15.39:0/3479572447 conn(0x7f98f00fa8c0 msgr2=0x7f98f00fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- 172.21.15.39:0/3479572447 shutdown_connections 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- 172.21.15.39:0/3479572447 wait complete. 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 Processor -- start 2024-02-19T19:10:03.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- start start 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98f00fed00 0x7f98f0196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 0x7f98f0196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f01970b0 con 0x7f98f0102280 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.689+0000 7f98f761f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f01971f0 con 0x7f98f00fed00 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 0x7f98f0196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 0x7f98f0196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54754/0 (socket says 172.21.15.39:54754) 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 -- 172.21.15.39:0/725679111 learned_addr learned my addr 172.21.15.39:0/725679111 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f53bb700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98f00fed00 0x7f98f0196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 -- 172.21.15.39:0/725679111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98f00fed00 msgr2=0x7f98f0196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:03.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98f00fed00 0x7f98f0196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 -- 172.21.15.39:0/725679111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e00095f0 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f4bba700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 0x7f98f0196a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f98e0000c00 tx=0x7f98e000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98e001d070 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f98e0005dc0 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f019bc40 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98e00226a0 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.690+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f019c130 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.691+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f0109810 con 0x7f98f0102280 2024-02-19T19:10:03.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.692+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f98e00038f0 con 0x7f98f0102280 2024-02-19T19:10:03.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.692+0000 7f98e67fc700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f98dc06d8a0 0x7f98dc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:03.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.692+0000 7f98f53bb700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f98dc06d8a0 0x7f98dc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:03.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.693+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f98e008e820 con 0x7f98f0102280 2024-02-19T19:10:03.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.693+0000 7f98f53bb700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f98dc06d8a0 0x7f98dc06fd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f98f0068b30 tx=0x7f98ec009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:03.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.697+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98e005bbd0 con 0x7f98f0102280 2024-02-19T19:10:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.991+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f98f0109a00 con 0x7f98f0102280 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.991+0000 7f98e67fc700 1 -- 172.21.15.39:0/725679111 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f98e005b760 con 0x7f98f0102280 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 5 2024-02-19T19:10:04.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 9 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:10:04.015 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f98dc06d8a0 msgr2=0x7f98dc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f98dc06d8a0 0x7f98dc06fd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f98f0068b30 tx=0x7f98ec009260 comp rx=0 tx=0).stop 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 msgr2=0x7f98f0196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 0x7f98f0196a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f98e0000c00 tx=0x7f98e000f690 comp rx=0 tx=0).stop 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 shutdown_connections 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f98dc06d8a0 0x7f98dc06fd60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98f00fed00 0x7f98f0196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 --2- 172.21.15.39:0/725679111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98f0102280 0x7f98f0196a90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.993+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 >> 172.21.15.39:0/725679111 conn(0x7f98f00fa8c0 msgr2=0x7f98f0105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.994+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 shutdown_connections 2024-02-19T19:10:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:03.994+0000 7f98f761f700 1 -- 172.21.15.39:0/725679111 wait complete. 2024-02-19T19:10:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 -- 172.21.15.39:0/1120809955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc100da0 msgr2=0x7f99fc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1120809955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc100da0 0x7f99fc101200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009910 tx=0x7f99f0009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 -- 172.21.15.39:0/1120809955 shutdown_connections 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1120809955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc100da0 0x7f99fc101200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1120809955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc0ffba0 0x7f99fc0fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 -- 172.21.15.39:0/1120809955 >> 172.21.15.39:0/1120809955 conn(0x7f99fc0fb120 msgr2=0x7f99fc0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 -- 172.21.15.39:0/1120809955 shutdown_connections 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.098+0000 7f9a00a63700 1 -- 172.21.15.39:0/1120809955 wait complete. 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f9a00a63700 1 Processor -- start 2024-02-19T19:10:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f9a00a63700 1 -- start start 2024-02-19T19:10:04.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f9a00a63700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 0x7f99fc069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f9a00a63700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc100da0 0x7f99fc19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f9a00a63700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc19b810 con 0x7f99fc0ffba0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f9a00a63700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc19b950 con 0x7f99fc100da0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 0x7f99fc069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 0x7f99fc069990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54770/0 (socket says 172.21.15.39:54770) 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 -- 172.21.15.39:0/1789059740 learned_addr learned my addr 172.21.15.39:0/1789059740 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99f9d9b700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc100da0 0x7f99fc19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 -- 172.21.15.39:0/1789059740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc100da0 msgr2=0x7f99fc19b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc100da0 0x7f99fc19b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 -- 172.21.15.39:0/1789059740 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00095f0 con 0x7f99fc0ffba0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99f9d9b700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc100da0 0x7f99fc19b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.099+0000 7f99fa59c700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 0x7f99fc069990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00d900 tx=0x7f99ec00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.100+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99ec0041f0 con 0x7f99fc0ffba0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.100+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f99ec004350 con 0x7f99fc0ffba0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.100+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99ec003d20 con 0x7f99fc0ffba0 2024-02-19T19:10:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.100+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc19bc30 con 0x7f99fc0ffba0 2024-02-19T19:10:04.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.100+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc19c130 con 0x7f99fc0ffba0 2024-02-19T19:10:04.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.101+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc108700 con 0x7f99fc0ffba0 2024-02-19T19:10:04.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.101+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f99ec009730 con 0x7f99fc0ffba0 2024-02-19T19:10:04.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.102+0000 7f99eb7fe700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f99e406d8a0 0x7f99e406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.102+0000 7f99f9d9b700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f99e406d8a0 0x7f99e406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.102+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f99ec08cde0 con 0x7f99fc0ffba0 2024-02-19T19:10:04.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.102+0000 7f99f9d9b700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f99e406d8a0 0x7f99e406fd60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f99fc069f10 tx=0x7f99f000b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:04.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.106+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99ec05a190 con 0x7f99fc0ffba0 2024-02-19T19:10:04.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.357+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f99fc108920 con 0x7f99fc0ffba0 2024-02-19T19:10:04.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.357+0000 7f99eb7fe700 1 -- 172.21.15.39:0/1789059740 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1868 (secure 0 0 0) 0x7f99ec003e80 con 0x7f99fc0ffba0 2024-02-19T19:10:04.379 INFO:teuthology.orchestra.run.smithi039.stdout:e15 2024-02-19T19:10:04.379 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:10:04.379 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 15 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:02:16.986275+0000 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:10:04.380 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 46 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:10:04.381 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{0:14498} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.39:6828/3608056816,v1:172.21.15.39:6829/3608056816] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:14510} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/438718767,v1:172.21.15.39:6827/438718767] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:04.382 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f99e406d8a0 msgr2=0x7f99e406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f99e406d8a0 0x7f99e406fd60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f99fc069f10 tx=0x7f99f000b560 comp rx=0 tx=0).stop 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 msgr2=0x7f99fc069990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 0x7f99fc069990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00d900 tx=0x7f99ec00dc10 comp rx=0 tx=0).stop 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 shutdown_connections 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f99e406d8a0 0x7f99e406fd60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99fc0ffba0 0x7f99fc069990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 --2- 172.21.15.39:0/1789059740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99fc100da0 0x7f99fc19b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 >> 172.21.15.39:0/1789059740 conn(0x7f99fc0fb120 msgr2=0x7f99fc103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 shutdown_connections 2024-02-19T19:10:04.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.359+0000 7f9a00a63700 1 -- 172.21.15.39:0/1789059740 wait complete. 2024-02-19T19:10:04.384 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 -- 172.21.15.39:0/876423241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 msgr2=0x7f0d440ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 --2- 172.21.15.39:0/876423241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d440ff380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009910 tx=0x7f0d34009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 -- 172.21.15.39:0/876423241 shutdown_connections 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 --2- 172.21.15.39:0/876423241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d44101f40 0x7f0d441023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 --2- 172.21.15.39:0/876423241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d440ff380 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 -- 172.21.15.39:0/876423241 >> 172.21.15.39:0/876423241 conn(0x7f0d440fa8c0 msgr2=0x7f0d440fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 -- 172.21.15.39:0/876423241 shutdown_connections 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.466+0000 7f0d4c581700 1 -- 172.21.15.39:0/876423241 wait complete. 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4c581700 1 Processor -- start 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4c581700 1 -- start start 2024-02-19T19:10:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4c581700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d441964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4c581700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d44101f40 0x7f0d441969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4c581700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d44197010 con 0x7f0d440fef60 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4c581700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d44197150 con 0x7f0d44101f40 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d441964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d441964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54798/0 (socket says 172.21.15.39:54798) 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 -- 172.21.15.39:0/2374650912 learned_addr learned my addr 172.21.15.39:0/2374650912 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d49b1c700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d44101f40 0x7f0d441969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 -- 172.21.15.39:0/2374650912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d44101f40 msgr2=0x7f0d441969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d44101f40 0x7f0d441969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 -- 172.21.15.39:0/2374650912 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d340095f0 con 0x7f0d440fef60 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.467+0000 7f0d4a31d700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d441964b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400bda0 tx=0x7f0d3400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.468+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d3401d070 con 0x7f0d440fef60 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.468+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0d3400f460 con 0x7f0d440fef60 2024-02-19T19:10:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.468+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4419bba0 con 0x7f0d440fef60 2024-02-19T19:10:04.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.468+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d34021620 con 0x7f0d440fef60 2024-02-19T19:10:04.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.468+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4419c090 con 0x7f0d440fef60 2024-02-19T19:10:04.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.468+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d44109800 con 0x7f0d440fef60 2024-02-19T19:10:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.469+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f0d3400f5d0 con 0x7f0d440fef60 2024-02-19T19:10:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.470+0000 7f0d437fe700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0d2c06d8a0 0x7f0d2c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.470+0000 7f0d49b1c700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0d2c06d8a0 0x7f0d2c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.470+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0d3408e7a0 con 0x7f0d440fef60 2024-02-19T19:10:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.470+0000 7f0d49b1c700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0d2c06d8a0 0x7f0d2c06fd60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005950 tx=0x7f0d3800b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:04.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.474+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d3405bad0 con 0x7f0d440fef60 2024-02-19T19:10:04.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:04.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:04.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: Upgrade: Updating osd.5 2024-02-19T19:10:04.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:10:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: Deploying daemon osd.5 on smithi086 2024-02-19T19:10:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:04 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/725679111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:04.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.696+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d44106890 con 0x7f0d2c06d8a0 2024-02-19T19:10:04.719 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:04.719 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:04.719 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: Upgrade: Updating osd.5 2024-02-19T19:10:04.719 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:04.719 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:10:04.720 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:04.720 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: Deploying daemon osd.5 on smithi086 2024-02-19T19:10:04.720 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:04 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/725679111' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:04.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.698+0000 7f0d437fe700 1 -- 172.21.15.39:0/2374650912 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f0d44106890 con 0x7f0d2c06d8a0 2024-02-19T19:10:04.720 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:10:04.720 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "11/23 daemons upgraded", 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:10:04.721 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0d2c06d8a0 msgr2=0x7f0d2c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0d2c06d8a0 0x7f0d2c06fd60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005950 tx=0x7f0d3800b410 comp rx=0 tx=0).stop 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 msgr2=0x7f0d441964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d441964b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400bda0 tx=0x7f0d3400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 shutdown_connections 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0d2c06d8a0 0x7f0d2c06fd60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d440fef60 0x7f0d441964b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 --2- 172.21.15.39:0/2374650912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d44101f40 0x7f0d441969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 >> 172.21.15.39:0/2374650912 conn(0x7f0d440fa8c0 msgr2=0x7f0d44105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.700+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 shutdown_connections 2024-02-19T19:10:04.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.701+0000 7f0d4c581700 1 -- 172.21.15.39:0/2374650912 wait complete. 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 -- 172.21.15.39:0/1036776162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c0ffbb0 msgr2=0x7fe68c0fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 --2- 172.21.15.39:0/1036776162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c0ffbb0 0x7fe68c0fffd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe67c009910 tx=0x7fe67c009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 -- 172.21.15.39:0/1036776162 shutdown_connections 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 --2- 172.21.15.39:0/1036776162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c100db0 0x7fe68c101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 --2- 172.21.15.39:0/1036776162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c0ffbb0 0x7fe68c0fffd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 -- 172.21.15.39:0/1036776162 >> 172.21.15.39:0/1036776162 conn(0x7fe68c0fb150 msgr2=0x7fe68c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 -- 172.21.15.39:0/1036776162 shutdown_connections 2024-02-19T19:10:04.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 -- 172.21.15.39:0/1036776162 wait complete. 2024-02-19T19:10:04.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 Processor -- start 2024-02-19T19:10:04.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.806+0000 7fe6926e7700 1 -- start start 2024-02-19T19:10:04.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe6926e7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0ffbb0 0x7fe68c069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe6926e7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 0x7fe68c19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe6926e7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c19b810 con 0x7fe68c100db0 2024-02-19T19:10:04.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe6926e7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c19b950 con 0x7fe68c0ffbb0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 0x7fe68c19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 0x7fe68c19b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54828/0 (socket says 172.21.15.39:54828) 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 -- 172.21.15.39:0/903938843 learned_addr learned my addr 172.21.15.39:0/903938843 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68bfff700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0ffbb0 0x7fe68c069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 -- 172.21.15.39:0/903938843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0ffbb0 msgr2=0x7fe68c069970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0ffbb0 0x7fe68c069970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 -- 172.21.15.39:0/903938843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0095f0 con 0x7fe68c100db0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe68b7fe700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 0x7fe68c19b240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe68000e920 tx=0x7fe68000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.807+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe68000c9d0 con 0x7fe68c100db0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.808+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c19bc30 con 0x7fe68c100db0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.808+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe680010430 con 0x7fe68c100db0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.808+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe680018650 con 0x7fe68c100db0 2024-02-19T19:10:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.808+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c19c180 con 0x7fe68c100db0 2024-02-19T19:10:04.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.809+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c108710 con 0x7fe68c100db0 2024-02-19T19:10:04.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.809+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fe6800105a0 con 0x7fe68c100db0 2024-02-19T19:10:04.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.809+0000 7fe6897fa700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe67406d8a0 0x7fe67406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:04.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.809+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fe680014070 con 0x7fe68c100db0 2024-02-19T19:10:04.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.809+0000 7fe68bfff700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe67406d8a0 0x7fe67406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:04.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.810+0000 7fe68bfff700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe67406d8a0 0x7fe67406fd60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00b6f0 tx=0x7fe67c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:04.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:04.814+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe680052d20 con 0x7fe68c100db0 2024-02-19T19:10:05.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.109+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe68c19c790 con 0x7fe68c100db0 2024-02-19T19:10:05.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.109+0000 7fe6897fa700 1 -- 172.21.15.39:0/903938843 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fe68005aad0 con 0x7fe68c100db0 2024-02-19T19:10:05.132 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:10:05.132 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:10:05.132 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe67406d8a0 msgr2=0x7fe67406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe67406d8a0 0x7fe67406fd60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00b6f0 tx=0x7fe67c005940 comp rx=0 tx=0).stop 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 msgr2=0x7fe68c19b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 0x7fe68c19b240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe68000e920 tx=0x7fe68000ec30 comp rx=0 tx=0).stop 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 shutdown_connections 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fe67406d8a0 0x7fe67406fd60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe68c0ffbb0 0x7fe68c069970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:05.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 --2- 172.21.15.39:0/903938843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe68c100db0 0x7fe68c19b240 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:05.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 >> 172.21.15.39:0/903938843 conn(0x7fe68c0fb150 msgr2=0x7fe68c103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:05.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 shutdown_connections 2024-02-19T19:10:05.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:05.112+0000 7fe6926e7700 1 -- 172.21.15.39:0/903938843 wait complete. 2024-02-19T19:10:05.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:05 smithi039 ceph-mon[92224]: pgmap v143: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:10:05.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:05 smithi039 ceph-mon[92224]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:05.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:05 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/1789059740' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:10:05.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:05 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/903938843' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:10:05.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:05 smithi086 ceph-mon[82675]: pgmap v143: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-02-19T19:10:05.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:05 smithi086 ceph-mon[82675]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:05.499 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:05 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/1789059740' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:10:05.500 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:05 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/903938843' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:10:06.678 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:06 smithi086 ceph-mon[82675]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:06.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:06 smithi039 ceph-mon[92224]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:06.951 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:06 smithi086 systemd[1]: Stopping Ceph osd.5 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:10:07.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:06 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[56563]: 2024-02-19T19:10:06.929+0000 7f7b6f0f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:10:07.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:06 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[56563]: 2024-02-19T19:10:06.929+0000 7f7b6f0f8700 -1 osd.5 74 *** Got signal Terminated *** 2024-02-19T19:10:07.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:06 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[56563]: 2024-02-19T19:10:06.929+0000 7f7b6f0f8700 -1 osd.5 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:10:07.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:07 smithi039 ceph-mon[92224]: pgmap v144: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:10:07.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:07 smithi039 ceph-mon[92224]: osd.5 marked itself down and dead 2024-02-19T19:10:07.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:07 smithi086 ceph-mon[82675]: pgmap v144: 65 pgs: 65 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:10:07.749 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:07 smithi086 ceph-mon[82675]: osd.5 marked itself down and dead 2024-02-19T19:10:08.035 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:07 smithi086 podman[102863]: 2024-02-19 19:10:07.728065426 +0000 UTC m=+0.904928505 container died 439153b12a3d6e89b9325157c2bbe657f7465f9ad286b92889e88e6be152dba3 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:10:08.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:08 smithi086 podman[102863]: 2024-02-19 19:10:08.104758813 +0000 UTC m=+1.281621895 container cleanup 439153b12a3d6e89b9325157c2bbe657f7465f9ad286b92889e88e6be152dba3 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T19:10:08.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:08 smithi086 bash[102863]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5 2024-02-19T19:10:08.740 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:08 smithi086 podman[102880]: 2024-02-19 19:10:08.381680688 +0000 UTC m=+0.652983165 container remove 439153b12a3d6e89b9325157c2bbe657f7465f9ad286b92889e88e6be152dba3 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-19T19:10:08.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:08 smithi086 ceph-mon[82675]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:10:08.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:08 smithi086 ceph-mon[82675]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-19T19:10:08.740 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:08 smithi086 ceph-mon[82675]: osdmap e75: 6 total, 5 up, 6 in 2024-02-19T19:10:08.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:08 smithi039 ceph-mon[92224]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:10:08.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:08 smithi039 ceph-mon[92224]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-19T19:10:08.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:08 smithi039 ceph-mon[92224]: osdmap e75: 6 total, 5 up, 6 in 2024-02-19T19:10:09.031 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:08 smithi086 podman[102973]: 2024-02-19 19:10:08.622492844 +0000 UTC m=+0.018257795 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:10:09.031 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:08 smithi086 podman[102973]: 2024-02-19 19:10:08.741082287 +0000 UTC m=+0.136847271 container create 4a04a89928f7ddf1f8bdc9be2284fc627a96e0ad3f447a56740126977ef86664 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-02-19T19:10:09.031 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 podman[102973]: 2024-02-19 19:10:09.010553809 +0000 UTC m=+0.406318751 container init 4a04a89928f7ddf1f8bdc9be2284fc627a96e0ad3f447a56740126977ef86664 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0) 2024-02-19T19:10:09.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 podman[102973]: 2024-02-19 19:10:09.013890462 +0000 UTC m=+0.409655406 container start 4a04a89928f7ddf1f8bdc9be2284fc627a96e0ad3f447a56740126977ef86664 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:10:09.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 podman[102973]: 2024-02-19 19:10:09.110458564 +0000 UTC m=+0.506223511 container attach 4a04a89928f7ddf1f8bdc9be2284fc627a96e0ad3f447a56740126977ef86664 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:10:09.634 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:09 smithi086 ceph-mon[82675]: pgmap v146: 65 pgs: 13 stale+active+clean, 52 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:10:09.634 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:09 smithi086 ceph-mon[82675]: osdmap e76: 6 total, 5 up, 6 in 2024-02-19T19:10:09.635 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 podman[102973]: 2024-02-19 19:10:09.305246356 +0000 UTC m=+0.701011307 container died 4a04a89928f7ddf1f8bdc9be2284fc627a96e0ad3f447a56740126977ef86664 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS) 2024-02-19T19:10:09.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:09 smithi039 ceph-mon[92224]: pgmap v146: 65 pgs: 13 stale+active+clean, 52 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-02-19T19:10:09.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:09 smithi039 ceph-mon[92224]: osdmap e76: 6 total, 5 up, 6 in 2024-02-19T19:10:10.286 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 podman[102999]: 2024-02-19 19:10:09.922937885 +0000 UTC m=+0.612677048 container remove 4a04a89928f7ddf1f8bdc9be2284fc627a96e0ad3f447a56740126977ef86664 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:10:10.287 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-19T19:10:10.287 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 systemd[1]: Stopped Ceph osd.5 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:10:10.287 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:09 smithi086 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.5.service: Consumed 5.078s CPU time. 2024-02-19T19:10:10.287 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:10 smithi086 systemd[1]: Starting Ceph osd.5 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T19:10:10.640 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:10 smithi086 ceph-mon[82675]: pgmap v148: 65 pgs: 3 active+undersized+degraded, 7 peering, 5 stale+active+clean, 10 active+undersized, 40 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 6/72 objects degraded (8.333%) 2024-02-19T19:10:10.640 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:10 smithi086 ceph-mon[82675]: Health check failed: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-02-19T19:10:10.640 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:10 smithi086 podman[103099]: 2024-02-19 19:10:10.263563956 +0000 UTC m=+0.018856211 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:10:10.640 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:10 smithi086 podman[103099]: 2024-02-19 19:10:10.382698574 +0000 UTC m=+0.137990819 container create d1e6be68fe8ed7daf73db38c6557af0574c60466ac9d8e2b1948b5b23b84374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:10:10.640 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:10 smithi086 podman[103099]: 2024-02-19 19:10:10.619118834 +0000 UTC m=+0.374411078 container init d1e6be68fe8ed7daf73db38c6557af0574c60466ac9d8e2b1948b5b23b84374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:10:10.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:10 smithi039 ceph-mon[92224]: pgmap v148: 65 pgs: 3 active+undersized+degraded, 7 peering, 5 stale+active+clean, 10 active+undersized, 40 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 6/72 objects degraded (8.333%) 2024-02-19T19:10:10.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:10 smithi039 ceph-mon[92224]: Health check failed: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-02-19T19:10:11.036 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:10 smithi086 podman[103099]: 2024-02-19 19:10:10.62232357 +0000 UTC m=+0.377615831 container start d1e6be68fe8ed7daf73db38c6557af0574c60466ac9d8e2b1948b5b23b84374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-19T19:10:11.036 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:10 smithi086 podman[103099]: 2024-02-19 19:10:10.710750961 +0000 UTC m=+0.466043212 container attach d1e6be68fe8ed7daf73db38c6557af0574c60466ac9d8e2b1948b5b23b84374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-02-19T19:10:11.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 bash[103099]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate[103114]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-19T19:10:11.413 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 podman[103099]: 2024-02-19 19:10:11.088734724 +0000 UTC m=+0.844026975 container died d1e6be68fe8ed7daf73db38c6557af0574c60466ac9d8e2b1948b5b23b84374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:10:11.974 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 podman[103235]: 2024-02-19 19:10:11.656711491 +0000 UTC m=+0.562911520 container remove d1e6be68fe8ed7daf73db38c6557af0574c60466ac9d8e2b1948b5b23b84374b (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-19T19:10:11.974 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 podman[103300]: 2024-02-19 19:10:11.769211476 +0000 UTC m=+0.017036548 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:10:11.974 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:11 smithi086 podman[103300]: 2024-02-19 19:10:11.874298856 +0000 UTC m=+0.122123921 container create 5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-19T19:10:12.224 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:12 smithi086 podman[103300]: 2024-02-19 19:10:12.118713654 +0000 UTC m=+0.366538733 container init 5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True) 2024-02-19T19:10:12.224 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:12 smithi086 podman[103300]: 2024-02-19 19:10:12.121658186 +0000 UTC m=+0.369483256 container start 5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:10:12.224 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:12 smithi086 ceph-osd[103318]: -- 172.21.15.86:0/298908363 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a7993611e0 con 0x55a799333000 2024-02-19T19:10:12.224 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:12 smithi086 bash[103300]: 5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400 2024-02-19T19:10:12.536 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:12 smithi086 systemd[1]: Started Ceph osd.5 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T19:10:13.036 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:12 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[103314]: 2024-02-19T19:10:12.663+0000 7fce120087c0 -1 Falling back to public interface 2024-02-19T19:10:13.401 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:13 smithi086 ceph-mon[82675]: pgmap v149: 65 pgs: 3 active+undersized+degraded, 7 peering, 5 stale+active+clean, 10 active+undersized, 40 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 6/72 objects degraded (8.333%) 2024-02-19T19:10:13.401 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:13.401 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:13.401 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:13 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:13 smithi039 ceph-mon[92224]: pgmap v149: 65 pgs: 3 active+undersized+degraded, 7 peering, 5 stale+active+clean, 10 active+undersized, 40 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 6/72 objects degraded (8.333%) 2024-02-19T19:10:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:13 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:14 smithi086 ceph-mon[82675]: pgmap v150: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-02-19T19:10:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:14.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:14.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:14 smithi039 ceph-mon[92224]: pgmap v150: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-02-19T19:10:14.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:14.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:16.403 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[103314]: 2024-02-19T19:10:16.119+0000 7fce120087c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-19T19:10:16.403 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[103314]: 2024-02-19T19:10:16.142+0000 7fce120087c0 -1 osd.5 74 log_to_monitors true 2024-02-19T19:10:16.730 INFO:journalctl@ceph.osd.5.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-5[103314]: 2024-02-19T19:10:16.413+0000 7fce03460700 -1 osd.5 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:10:16.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:16.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:16.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-mon[82675]: from='osd.5 [v2:172.21.15.86:6816/3455419941,v1:172.21.15.86:6817/3455419941]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:10:16.730 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:16 smithi086 ceph-mon[82675]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:10:16.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:16 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:16.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:16 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:16.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:16 smithi039 ceph-mon[92224]: from='osd.5 [v2:172.21.15.86:6816/3455419941,v1:172.21.15.86:6817/3455419941]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:10:16.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:16 smithi039 ceph-mon[92224]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:10:17.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: pgmap v151: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 377 B/s rd, 0 op/s; 8/72 objects degraded (11.111%) 2024-02-19T19:10:17.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T19:10:17.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: osdmap e77: 6 total, 5 up, 6 in 2024-02-19T19:10:17.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: from='osd.5 [v2:172.21.15.86:6816/3455419941,v1:172.21.15.86:6817/3455419941]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:10:17.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:10:17.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:17.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:17 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:17.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: pgmap v151: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 591 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 377 B/s rd, 0 op/s; 8/72 objects degraded (11.111%) 2024-02-19T19:10:17.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T19:10:17.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: osdmap e77: 6 total, 5 up, 6 in 2024-02-19T19:10:17.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: from='osd.5 [v2:172.21.15.86:6816/3455419941,v1:172.21.15.86:6817/3455419941]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:10:17.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-02-19T19:10:17.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:17.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:17 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:18.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:18 smithi086 ceph-mon[82675]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:10:18.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:18 smithi086 ceph-mon[82675]: osd.5 [v2:172.21.15.86:6816/3455419941,v1:172.21.15.86:6817/3455419941] boot 2024-02-19T19:10:18.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:18 smithi086 ceph-mon[82675]: osdmap e78: 6 total, 6 up, 6 in 2024-02-19T19:10:18.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:18 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:10:18.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:18 smithi086 ceph-mon[82675]: pgmap v154: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 8/72 objects degraded (11.111%) 2024-02-19T19:10:18.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:18 smithi039 ceph-mon[92224]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:10:18.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:18 smithi039 ceph-mon[92224]: osd.5 [v2:172.21.15.86:6816/3455419941,v1:172.21.15.86:6817/3455419941] boot 2024-02-19T19:10:18.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:18 smithi039 ceph-mon[92224]: osdmap e78: 6 total, 6 up, 6 in 2024-02-19T19:10:18.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:18 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:10:18.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:18 smithi039 ceph-mon[92224]: pgmap v154: 65 pgs: 5 active+undersized+degraded, 23 active+undersized, 37 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 639 B/s rd, 1 op/s; 8/72 objects degraded (11.111%) 2024-02-19T19:10:19.625 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:19 smithi086 ceph-mon[82675]: osdmap e79: 6 total, 6 up, 6 in 2024-02-19T19:10:19.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:19 smithi039 ceph-mon[92224]: osdmap e79: 6 total, 6 up, 6 in 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: pgmap v156: 65 pgs: 3 peering, 16 active+undersized, 46 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:10:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: Upgrade: Setting container_image for all osd 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-19T19:10:20.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: Upgrade: Setting require_osd_release to 19 squid 2024-02-19T19:10:20.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-19T19:10:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: pgmap v156: 65 pgs: 3 peering, 16 active+undersized, 46 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:10:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:10:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: Upgrade: Setting container_image for all osd 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-19T19:10:20.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-19T19:10:20.788 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-19T19:10:20.788 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-19T19:10:20.788 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-19T19:10:20.788 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-19T19:10:20.788 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: Upgrade: Setting require_osd_release to 19 squid 2024-02-19T19:10:20.788 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-19T19:10:21.590 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded) 2024-02-19T19:10:21.591 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-19T19:10:21.591 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-19T19:10:21.591 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: osdmap e80: 6 total, 6 up, 6 in 2024-02-19T19:10:21.591 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-02-19T19:10:21.591 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:21.591 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:21 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-02-19T19:10:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded) 2024-02-19T19:10:21.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-19T19:10:21.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-19T19:10:21.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: osdmap e80: 6 total, 6 up, 6 in 2024-02-19T19:10:21.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-02-19T19:10:21.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:21.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:21 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-02-19T19:10:22.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: osdmap e81: 6 total, 6 up, 6 in 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: pgmap v159: 65 pgs: 3 peering, 3 active+undersized, 59 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1005 B/s rd, 1 op/s 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 2 up:standby 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi039.omsmsr"]}]: dispatch 2024-02-19T19:10:22.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:22 smithi039 ceph-mon[92224]: Upgrade: It appears safe to stop mds.cephfs.smithi039.omsmsr 2024-02-19T19:10:22.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: osdmap e81: 6 total, 6 up, 6 in 2024-02-19T19:10:22.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: pgmap v159: 65 pgs: 3 peering, 3 active+undersized, 59 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1005 B/s rd, 1 op/s 2024-02-19T19:10:22.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-02-19T19:10:22.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 2 up:standby 2024-02-19T19:10:22.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:22.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi039.omsmsr"]}]: dispatch 2024-02-19T19:10:22.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:22 smithi086 ceph-mon[82675]: Upgrade: It appears safe to stop mds.cephfs.smithi039.omsmsr 2024-02-19T19:10:24.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:24 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6828/3305800521,v1:172.21.15.39:6829/3305800521] up:boot 2024-02-19T19:10:24.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:24 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:10:24.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:24 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:10:24.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:24 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:24.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:24 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:10:24.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:24 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:24.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:24 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6828/3305800521,v1:172.21.15.39:6829/3305800521] up:boot 2024-02-19T19:10:24.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:24 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:10:24.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:24 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:10:24.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:24 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:24.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:24 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.omsmsr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:10:24.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:24 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:25.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:25 smithi039 ceph-mon[92224]: pgmap v160: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2024-02-19T19:10:25.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:25 smithi039 ceph-mon[92224]: Upgrade: Updating mds.cephfs.smithi039.omsmsr 2024-02-19T19:10:25.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:25 smithi039 ceph-mon[92224]: Deploying daemon mds.cephfs.smithi039.omsmsr on smithi039 2024-02-19T19:10:25.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:25 smithi086 ceph-mon[82675]: pgmap v160: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2024-02-19T19:10:25.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:25 smithi086 ceph-mon[82675]: Upgrade: Updating mds.cephfs.smithi039.omsmsr 2024-02-19T19:10:25.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:25 smithi086 ceph-mon[82675]: Deploying daemon mds.cephfs.smithi039.omsmsr on smithi039 2024-02-19T19:10:27.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:27 smithi086 ceph-mon[82675]: pgmap v161: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1014 B/s rd, 1 op/s 2024-02-19T19:10:27.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:27 smithi039 ceph-mon[92224]: pgmap v161: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1014 B/s rd, 1 op/s 2024-02-19T19:10:28.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:28 smithi086 ceph-mon[82675]: osdmap e82: 6 total, 6 up, 6 in 2024-02-19T19:10:28.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:28 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 2 up:standby 2024-02-19T19:10:28.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:28 smithi039 ceph-mon[92224]: osdmap e82: 6 total, 6 up, 6 in 2024-02-19T19:10:28.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:28 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 2 up:standby 2024-02-19T19:10:29.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:29 smithi086 ceph-mon[82675]: pgmap v163: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 490 B/s rd, 0 op/s 2024-02-19T19:10:29.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:29.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:29.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:29.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:29 smithi039 ceph-mon[92224]: pgmap v163: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 490 B/s rd, 0 op/s 2024-02-19T19:10:29.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:29.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:29.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:30.354 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:10:30.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:10:31.439 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:31 smithi039 ceph-mon[92224]: pgmap v164: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 381 B/s rd, 0 op/s 2024-02-19T19:10:31.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:31 smithi086 ceph-mon[82675]: pgmap v164: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 381 B/s rd, 0 op/s 2024-02-19T19:10:32.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: pgmap v165: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2024-02-19T19:10:32.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:32.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:32.954 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:32.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:boot 2024-02-19T19:10:32.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:10:32.955 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:10:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: pgmap v165: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2024-02-19T19:10:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:33.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:boot 2024-02-19T19:10:33.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:10:33.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:10:35.132 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:34 smithi039 ceph-mon[92224]: pgmap v166: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- 172.21.15.39:0/3531468176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100e10 msgr2=0x7f6118101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 --2- 172.21.15.39:0/3531468176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100e10 0x7f6118101270 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6104009910 tx=0x7f6104009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- 172.21.15.39:0/3531468176 shutdown_connections 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 --2- 172.21.15.39:0/3531468176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100e10 0x7f6118101270 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 --2- 172.21.15.39:0/3531468176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61180ffc10 0x7f6118100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- 172.21.15.39:0/3531468176 >> 172.21.15.39:0/3531468176 conn(0x7f61180fb190 msgr2=0x7f61180fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- 172.21.15.39:0/3531468176 shutdown_connections 2024-02-19T19:10:35.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- 172.21.15.39:0/3531468176 wait complete. 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 Processor -- start 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- start start 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 0x7f6118198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6118100e10 0x7f6118198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61181991e0 con 0x7f61180ffc10 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.228+0000 7f611ee05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6118199320 con 0x7f6118100e10 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 0x7f6118198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 0x7f6118198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51344/0 (socket says 172.21.15.39:51344) 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 -- 172.21.15.39:0/2808741333 learned_addr learned my addr 172.21.15.39:0/2808741333 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f6117fff700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6118100e10 0x7f6118198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 -- 172.21.15.39:0/2808741333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6118100e10 msgr2=0x7f6118198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6118100e10 0x7f6118198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 -- 172.21.15.39:0/2808741333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61040095f0 con 0x7f61180ffc10 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f6117fff700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6118100e10 0x7f6118198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611cba1700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 0x7f6118198680 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f610c00d900 tx=0x7f610c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610c0041f0 con 0x7f61180ffc10 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f610c004350 con 0x7f61180ffc10 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611819ddd0 con 0x7f61180ffc10 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f610c003da0 con 0x7f61180ffc10 2024-02-19T19:10:35.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.229+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611819e2a0 con 0x7f61180ffc10 2024-02-19T19:10:35.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.230+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6118108700 con 0x7f61180ffc10 2024-02-19T19:10:35.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.231+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f610c009730 con 0x7f61180ffc10 2024-02-19T19:10:35.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.231+0000 7f6115ffb700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f610806d8a0 0x7f610806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.231+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f610c08d5c0 con 0x7f61180ffc10 2024-02-19T19:10:35.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.231+0000 7f6117fff700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f610806d8a0 0x7f610806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.232+0000 7f6117fff700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f610806d8a0 0x7f610806fd60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f610400b5c0 tx=0x7f6104005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:35.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.236+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f610c05a970 con 0x7f61180ffc10 2024-02-19T19:10:35.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:34 smithi086 ceph-mon[82675]: pgmap v166: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2024-02-19T19:10:35.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.461+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6118068c90 con 0x7f610806d8a0 2024-02-19T19:10:35.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.462+0000 7f6115ffb700 1 -- 172.21.15.39:0/2808741333 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6118068c90 con 0x7f610806d8a0 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f610806d8a0 msgr2=0x7f610806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f610806d8a0 0x7f610806fd60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f610400b5c0 tx=0x7f6104005e20 comp rx=0 tx=0).stop 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 msgr2=0x7f6118198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 0x7f6118198680 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f610c00d900 tx=0x7f610c00dcc0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 shutdown_connections 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f610806d8a0 0x7f610806fd60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61180ffc10 0x7f6118198680 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 --2- 172.21.15.39:0/2808741333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6118100e10 0x7f6118198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 >> 172.21.15.39:0/2808741333 conn(0x7f61180fb190 msgr2=0x7f6118068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.464+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 shutdown_connections 2024-02-19T19:10:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.465+0000 7f611ee05700 1 -- 172.21.15.39:0/2808741333 wait complete. 2024-02-19T19:10:35.496 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:10:35.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.571+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/2852313098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 msgr2=0x7f1e18106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.571+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/2852313098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e18106480 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009910 tx=0x7f1e0c009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.571+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/2852313098 shutdown_connections 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.571+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/2852313098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e18106480 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.571+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/2852313098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e18101760 0x7f1e18103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.571+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/2852313098 >> 172.21.15.39:0/2852313098 conn(0x7f1e180fb150 msgr2=0x7f1e180fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/2852313098 shutdown_connections 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/2852313098 wait complete. 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 Processor -- start 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 -- start start 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e18101760 0x7f1e181010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e180ff710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180ffc50 con 0x7f1e18104090 2024-02-19T19:10:35.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.572+0000 7f1e1cd6a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180ffd90 con 0x7f1e18101760 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e180ff710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e180ff710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51362/0 (socket says 172.21.15.39:51362) 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 -- 172.21.15.39:0/4089222785 learned_addr learned my addr 172.21.15.39:0/4089222785 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e1659c700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e18101760 0x7f1e181010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 -- 172.21.15.39:0/4089222785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e18101760 msgr2=0x7f1e181010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e18101760 0x7f1e181010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 -- 172.21.15.39:0/4089222785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c0095f0 con 0x7f1e18104090 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e15d9b700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e180ff710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00fa00 tx=0x7f1e0c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e0c01d070 con 0x7f1e18104090 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e18100010 con 0x7f1e18104090 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1e0c00fe90 con 0x7f1e18104090 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e0c0226a0 con 0x7f1e18104090 2024-02-19T19:10:35.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.573+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e18100500 con 0x7f1e18104090 2024-02-19T19:10:35.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.574+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e18190640 con 0x7f1e18104090 2024-02-19T19:10:35.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.575+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f1e0c003680 con 0x7f1e18104090 2024-02-19T19:10:35.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.575+0000 7f1e077fe700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1e0006d8a0 0x7f1e0006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.575+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1e0c08e920 con 0x7f1e18104090 2024-02-19T19:10:35.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.575+0000 7f1e1659c700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1e0006d8a0 0x7f1e0006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.576+0000 7f1e1659c700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1e0006d8a0 0x7f1e0006fd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009c60 tx=0x7f1e08009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:35.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.580+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e0c05bc50 con 0x7f1e18104090 2024-02-19T19:10:35.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.818+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e180611d0 con 0x7f1e0006d8a0 2024-02-19T19:10:35.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.819+0000 7f1e077fe700 1 -- 172.21.15.39:0/4089222785 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1e180611d0 con 0x7f1e0006d8a0 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1e0006d8a0 msgr2=0x7f1e0006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1e0006d8a0 0x7f1e0006fd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009c60 tx=0x7f1e08009260 comp rx=0 tx=0).stop 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 msgr2=0x7f1e180ff710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e180ff710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00fa00 tx=0x7f1e0c00fa30 comp rx=0 tx=0).stop 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 shutdown_connections 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1e0006d8a0 0x7f1e0006fd60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.821+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e18101760 0x7f1e181010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.822+0000 7f1e1cd6a700 1 --2- 172.21.15.39:0/4089222785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e18104090 0x7f1e180ff710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.822+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 >> 172.21.15.39:0/4089222785 conn(0x7f1e180fb150 msgr2=0x7f1e180fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:35.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.822+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 shutdown_connections 2024-02-19T19:10:35.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.822+0000 7f1e1cd6a700 1 -- 172.21.15.39:0/4089222785 wait complete. 2024-02-19T19:10:35.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 -- 172.21.15.39:0/273788358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 msgr2=0x7fd304103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 --2- 172.21.15.39:0/273788358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304103b90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd300009910 tx=0x7fd300009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 -- 172.21.15.39:0/273788358 shutdown_connections 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 --2- 172.21.15.39:0/273788358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3041040d0 0x7fd3041064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 --2- 172.21.15.39:0/273788358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304103b90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 -- 172.21.15.39:0/273788358 >> 172.21.15.39:0/273788358 conn(0x7fd3040fb190 msgr2=0x7fd3040fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 -- 172.21.15.39:0/273788358 shutdown_connections 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.927+0000 7fd30c358700 1 -- 172.21.15.39:0/273788358 wait complete. 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30c358700 1 Processor -- start 2024-02-19T19:10:35.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30c358700 1 -- start start 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30c358700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30c358700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3041040d0 0x7fd3041969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30c358700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304196fe0 con 0x7fd3041017a0 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30c358700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304197120 con 0x7fd3041040d0 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30a0f4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30a0f4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51374/0 (socket says 172.21.15.39:51374) 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30a0f4700 1 -- 172.21.15.39:0/3563515480 learned_addr learned my addr 172.21.15.39:0/3563515480 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd3098f3700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3041040d0 0x7fd3041969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30a0f4700 1 -- 172.21.15.39:0/3563515480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3041040d0 msgr2=0x7fd3041969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30a0f4700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3041040d0 0x7fd3041969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:35.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.928+0000 7fd30a0f4700 1 -- 172.21.15.39:0/3563515480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000095f0 con 0x7fd3041017a0 2024-02-19T19:10:35.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.929+0000 7fd30a0f4700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304196480 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd30000bd70 tx=0x7fd30000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:35.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.929+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd30001d070 con 0x7fd3041017a0 2024-02-19T19:10:35.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.929+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd300005940 con 0x7fd3041017a0 2024-02-19T19:10:35.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.929+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30419bb70 con 0x7fd3041017a0 2024-02-19T19:10:35.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.929+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd300021430 con 0x7fd3041017a0 2024-02-19T19:10:35.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.929+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30419c060 con 0x7fd3041017a0 2024-02-19T19:10:35.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.930+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd304190630 con 0x7fd3041017a0 2024-02-19T19:10:35.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.931+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fd300003a60 con 0x7fd3041017a0 2024-02-19T19:10:35.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.932+0000 7fd2fb7fe700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd2f006d8a0 0x7fd2f006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:35.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.932+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd30008e860 con 0x7fd3041017a0 2024-02-19T19:10:35.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.932+0000 7fd3098f3700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd2f006d8a0 0x7fd2f006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:35.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.933+0000 7fd3098f3700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd2f006d8a0 0x7fd2f006fd60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4005950 tx=0x7fd2f400a300 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:35.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:35.938+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd30005baf0 con 0x7fd3041017a0 2024-02-19T19:10:36.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.162+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3040611d0 con 0x7fd2f006d8a0 2024-02-19T19:10:36.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.173+0000 7fd2fb7fe700 1 -- 172.21.15.39:0/3563515480 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd3040611d0 con 0x7fd2f006d8a0 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (12m) 107s ago 14m 24.8M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (14m) 107s ago 14m 8388k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (13m) 20s ago 13m 8442k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (3m) 107s ago 14m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (3m) 20s ago 13m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (12m) 107s ago 13m 88.3M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (8m) 107s ago 8m 26.4M - 18.2.1 94eadd990e0a 66aca9f9b4f3 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 starting - - - - 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (8m) 20s ago 8m 29.2M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (8m) 20s ago 8m 17.9M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:10:36.195 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (5m) 107s ago 15m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (4m) 20s ago 13m 450M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (4m) 107s ago 15m 46.2M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (4m) 20s ago 13m 36.7M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (14m) 107s ago 14m 15.6M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (13m) 20s ago 13m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (2m) 107s ago 12m 51.9M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (2m) 107s ago 11m 51.3M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 546fbde6ffab 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (114s) 107s ago 11m 11.7M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 3d2286828d68 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (92s) 20s ago 10m 44.9M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 37ce649da9a6 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (58s) 20s ago 10m 45.9M 2240M 19.0.0-1450-gf78a58c0 f081f847070a f6b44d6c5641 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (24s) 20s ago 9m 12.7M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 5b366510b403 2024-02-19T19:10:36.196 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (4m) 107s ago 13m 62.3M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd2f006d8a0 msgr2=0x7fd2f006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd2f006d8a0 0x7fd2f006fd60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4005950 tx=0x7fd2f400a300 comp rx=0 tx=0).stop 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 msgr2=0x7fd304196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304196480 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd30000bd70 tx=0x7fd30000bda0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 shutdown_connections 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fd2f006d8a0 0x7fd2f006fd60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3041017a0 0x7fd304196480 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 --2- 172.21.15.39:0/3563515480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3041040d0 0x7fd3041969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 >> 172.21.15.39:0/3563515480 conn(0x7fd3040fb190 msgr2=0x7fd3040fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:36.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 shutdown_connections 2024-02-19T19:10:36.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.175+0000 7fd30c358700 1 -- 172.21.15.39:0/3563515480 wait complete. 2024-02-19T19:10:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.278+0000 7f147caa7700 1 -- 172.21.15.39:0/94371511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ff520 msgr2=0x7f14780ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.278+0000 7f147caa7700 1 --2- 172.21.15.39:0/94371511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ff520 0x7f14780ff940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1468009910 tx=0x7f1468009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 -- 172.21.15.39:0/94371511 shutdown_connections 2024-02-19T19:10:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 --2- 172.21.15.39:0/94371511 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780ffe80 0x7f1478100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 --2- 172.21.15.39:0/94371511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ff520 0x7f14780ff940 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 -- 172.21.15.39:0/94371511 >> 172.21.15.39:0/94371511 conn(0x7f14780fb120 msgr2=0x7f14780fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 -- 172.21.15.39:0/94371511 shutdown_connections 2024-02-19T19:10:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 -- 172.21.15.39:0/94371511 wait complete. 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 Processor -- start 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.279+0000 7f147caa7700 1 -- start start 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147caa7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780ff520 0x7f1478198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147caa7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 0x7f1478198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147caa7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14781991d0 con 0x7f14780ffe80 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147caa7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478199310 con 0x7f14780ff520 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 0x7f1478198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 0x7f1478198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51400/0 (socket says 172.21.15.39:51400) 2024-02-19T19:10:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 -- 172.21.15.39:0/2570882624 learned_addr learned my addr 172.21.15.39:0/2570882624 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147659c700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780ff520 0x7f1478198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 -- 172.21.15.39:0/2570882624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780ff520 msgr2=0x7f1478198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780ff520 0x7f1478198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 -- 172.21.15.39:0/2570882624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14680095f0 con 0x7f14780ffe80 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f1475d9b700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 0x7f1478198bb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f146c00d8d0 tx=0x7f146c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c0098a0 con 0x7f14780ffe80 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f146c010460 con 0x7f14780ffe80 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147819ddc0 con 0x7f14780ffe80 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c00f5f0 con 0x7f14780ffe80 2024-02-19T19:10:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.280+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147819e2e0 con 0x7f14780ffe80 2024-02-19T19:10:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.281+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147810b9f0 con 0x7f14780ffe80 2024-02-19T19:10:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.282+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f146c009a00 con 0x7f14780ffe80 2024-02-19T19:10:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.282+0000 7f14677fe700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f146006d8a0 0x7f146006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.282+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f146c08d2c0 con 0x7f14780ffe80 2024-02-19T19:10:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.282+0000 7f147659c700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f146006d8a0 0x7f146006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.283+0000 7f147659c700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f146006d8a0 0x7f146006fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f146800b5c0 tx=0x7f1468005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:36.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.287+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f146c05a670 con 0x7f14780ffe80 2024-02-19T19:10:36.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.590+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f147810bbb0 con 0x7f14780ffe80 2024-02-19T19:10:36.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.590+0000 7f14677fe700 1 -- 172.21.15.39:0/2570882624 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f146c05a200 con 0x7f14780ffe80 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-02-19T19:10:36.612 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:10:36.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:10:36.613 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:10:36.613 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-02-19T19:10:36.613 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 11 2024-02-19T19:10:36.613 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:10:36.613 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f146006d8a0 msgr2=0x7f146006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f146006d8a0 0x7f146006fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f146800b5c0 tx=0x7f1468005ee0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 msgr2=0x7f1478198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 0x7f1478198bb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f146c00d8d0 tx=0x7f146c00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 shutdown_connections 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f146006d8a0 0x7f146006fd60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14780ff520 0x7f1478198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 --2- 172.21.15.39:0/2570882624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14780ffe80 0x7f1478198bb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 >> 172.21.15.39:0/2570882624 conn(0x7f14780fb120 msgr2=0x7f1478107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 shutdown_connections 2024-02-19T19:10:36.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.592+0000 7f147caa7700 1 -- 172.21.15.39:0/2570882624 wait complete. 2024-02-19T19:10:36.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 -- 172.21.15.39:0/1040739408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 msgr2=0x7f71bc0fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 --2- 172.21.15.39:0/1040739408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc0fffe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009910 tx=0x7f71ac009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:36.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 -- 172.21.15.39:0/1040739408 shutdown_connections 2024-02-19T19:10:36.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 --2- 172.21.15.39:0/1040739408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc100dc0 0x7f71bc101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 --2- 172.21.15.39:0/1040739408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc0fffe0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 -- 172.21.15.39:0/1040739408 >> 172.21.15.39:0/1040739408 conn(0x7f71bc0fb160 msgr2=0x7f71bc0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:36.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 -- 172.21.15.39:0/1040739408 shutdown_connections 2024-02-19T19:10:36.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 -- 172.21.15.39:0/1040739408 wait complete. 2024-02-19T19:10:36.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 Processor -- start 2024-02-19T19:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.697+0000 7f71c1db2700 1 -- start start 2024-02-19T19:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71c1db2700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71c1db2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc100dc0 0x7f71bc19d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71c1db2700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc197340 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71c1db2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc19de10 con 0x7f71bc100dc0 2024-02-19T19:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc069970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34408/0 (socket says 172.21.15.39:34408) 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 -- 172.21.15.39:0/840202126 learned_addr learned my addr 172.21.15.39:0/840202126 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71baffd700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc100dc0 0x7f71bc19d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 -- 172.21.15.39:0/840202126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc100dc0 msgr2=0x7f71bc19d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc100dc0 0x7f71bc19d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 -- 172.21.15.39:0/840202126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0095f0 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71bb7fe700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc069970 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bdd0 tx=0x7f71ac00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71ac01d070 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f71ac00f460 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc19e060 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71ac017630 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.698+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc19e550 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.700+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc18f530 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.700+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f71ac00f5d0 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.700+0000 7f71b8ff9700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f71a406d8a0 0x7f71a406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.700+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f71ac08ea30 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.700+0000 7f71baffd700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f71a406d8a0 0x7f71a406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.701+0000 7f71baffd700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f71a406d8a0 0x7f71a406fd60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f71bc069ef0 tx=0x7f71b000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:36.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.705+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f71ac05bd60 con 0x7f71bc0ffbc0 2024-02-19T19:10:36.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:36 smithi039 ceph-mon[92224]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:36.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:36 smithi039 ceph-mon[92224]: pgmap v167: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:36.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:36 smithi039 ceph-mon[92224]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:36.924 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:36 smithi039 ceph-mon[92224]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:36.925 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:36 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/2570882624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:37.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.984+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f71bc068af0 con 0x7f71bc0ffbc0 2024-02-19T19:10:37.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.984+0000 7f71b8ff9700 1 -- 172.21.15.39:0/840202126 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 19 v19) v1 ==== 76+0+1840 (secure 0 0 0) 0x7f71ac05b8f0 con 0x7f71bc0ffbc0 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:e19 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 16 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:10:21.225951+0000 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:10:37.006 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 0 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24267} 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:10:37.007 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{0:24267} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.86:6824/2684130153,v1:172.21.15.86:6825/2684130153] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:24279} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{-1:34376} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6828/3305800521,v1:172.21.15.39:6829/3305800521] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:10:37.008 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f71a406d8a0 msgr2=0x7f71a406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f71a406d8a0 0x7f71a406fd60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f71bc069ef0 tx=0x7f71b000b410 comp rx=0 tx=0).stop 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 msgr2=0x7f71bc069970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc069970 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00bdd0 tx=0x7f71ac00be00 comp rx=0 tx=0).stop 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 shutdown_connections 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f71a406d8a0 0x7f71a406fd60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71bc0ffbc0 0x7f71bc069970 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 --2- 172.21.15.39:0/840202126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71bc100dc0 0x7f71bc19d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 >> 172.21.15.39:0/840202126 conn(0x7f71bc0fb160 msgr2=0x7f71bc1040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:37.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 shutdown_connections 2024-02-19T19:10:37.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:36.987+0000 7f71c1db2700 1 -- 172.21.15.39:0/840202126 wait complete. 2024-02-19T19:10:37.010 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 19 2024-02-19T19:10:37.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.093+0000 7fa76a4ee700 1 -- 172.21.15.39:0/3279870503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 msgr2=0x7fa7640fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/3279870503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7640fffe0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa754009910 tx=0x7fa754009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:37.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 -- 172.21.15.39:0/3279870503 shutdown_connections 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/3279870503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 0x7fa764101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/3279870503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7640fffe0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 -- 172.21.15.39:0/3279870503 >> 172.21.15.39:0/3279870503 conn(0x7fa7640fb160 msgr2=0x7fa7640fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 -- 172.21.15.39:0/3279870503 shutdown_connections 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 -- 172.21.15.39:0/3279870503 wait complete. 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 Processor -- start 2024-02-19T19:10:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.094+0000 7fa76a4ee700 1 -- start start 2024-02-19T19:10:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa76a4ee700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7641957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa76a4ee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 0x7fa764195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa76a4ee700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764196350 con 0x7fa7640ffbc0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa76a4ee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa764196490 con 0x7fa764100dc0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7641957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7641957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34414/0 (socket says 172.21.15.39:34414) 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 -- 172.21.15.39:0/1955251075 learned_addr learned my addr 172.21.15.39:0/1955251075 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa7637fe700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 0x7fa764195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 -- 172.21.15.39:0/1955251075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 msgr2=0x7fa764195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 0x7fa764195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 -- 172.21.15.39:0/1955251075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7540095f0 con 0x7fa7640ffbc0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa7637fe700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 0x7fa764195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa763fff700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7641957f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa75400bd70 tx=0x7fa75400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75401d070 con 0x7fa7640ffbc0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa754022470 con 0x7fa7640ffbc0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76419aee0 con 0x7fa7640ffbc0 2024-02-19T19:10:37.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.095+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa75400f690 con 0x7fa7640ffbc0 2024-02-19T19:10:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.096+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76419b350 con 0x7fa7640ffbc0 2024-02-19T19:10:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.097+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76418f5b0 con 0x7fa7640ffbc0 2024-02-19T19:10:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.098+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fa7540225e0 con 0x7fa7640ffbc0 2024-02-19T19:10:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.098+0000 7fa7617fa700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa74c06d8a0 0x7fa74c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.098+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa75408ed00 con 0x7fa7640ffbc0 2024-02-19T19:10:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.098+0000 7fa7637fe700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa74c06d8a0 0x7fa74c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.099+0000 7fa7637fe700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa74c06d8a0 0x7fa74c06fd60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa758007900 tx=0x7fa758008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:37.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.103+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa754092030 con 0x7fa7640ffbc0 2024-02-19T19:10:37.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:36 smithi086 ceph-mon[82675]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:37.247 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:36 smithi086 ceph-mon[82675]: pgmap v167: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:37.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:36 smithi086 ceph-mon[82675]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:37.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:36 smithi086 ceph-mon[82675]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:37.248 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:36 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/2570882624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:37.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.336+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7640611d0 con 0x7fa74c06d8a0 2024-02-19T19:10:37.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.337+0000 7fa7617fa700 1 -- 172.21.15.39:0/1955251075 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa7640611d0 con 0x7fa74c06d8a0 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "osd", 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:10:37.359 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "12/23 daemons upgraded", 2024-02-19T19:10:37.360 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading mds daemons", 2024-02-19T19:10:37.360 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:10:37.360 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:10:37.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa74c06d8a0 msgr2=0x7fa74c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa74c06d8a0 0x7fa74c06fd60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa758007900 tx=0x7fa758008040 comp rx=0 tx=0).stop 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 msgr2=0x7fa7641957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7641957f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa75400bd70 tx=0x7fa75400be50 comp rx=0 tx=0).stop 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 shutdown_connections 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa74c06d8a0 0x7fa74c06fd60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7640ffbc0 0x7fa7641957f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 --2- 172.21.15.39:0/1955251075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa764100dc0 0x7fa764195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 >> 172.21.15.39:0/1955251075 conn(0x7fa7640fb160 msgr2=0x7fa7641041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 shutdown_connections 2024-02-19T19:10:37.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.340+0000 7fa76a4ee700 1 -- 172.21.15.39:0/1955251075 wait complete. 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/2360379452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4104070 msgr2=0x7f2ed4106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/2360379452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4104070 0x7f2ed4106460 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009910 tx=0x7f2ec8009c20 comp rx=0 tx=0).stop 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/2360379452 shutdown_connections 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/2360379452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4104070 0x7f2ed4106460 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/2360379452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4101740 0x7f2ed4103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/2360379452 >> 172.21.15.39:0/2360379452 conn(0x7f2ed40fb150 msgr2=0x7f2ed40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/2360379452 shutdown_connections 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.454+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/2360379452 wait complete. 2024-02-19T19:10:37.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed9f1f700 1 Processor -- start 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed9f1f700 1 -- start start 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed9f1f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 0x7f2ed41010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed9f1f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4104070 0x7f2ed40ff6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed9f1f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40ffc30 con 0x7f2ed4101740 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed9f1f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40ffd70 con 0x7f2ed4104070 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 0x7f2ed41010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 0x7f2ed41010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34438/0 (socket says 172.21.15.39:34438) 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed37fe700 1 -- 172.21.15.39:0/3763511173 learned_addr learned my addr 172.21.15.39:0/3763511173 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.455+0000 7f2ed2ffd700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4104070 0x7f2ed40ff6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed37fe700 1 -- 172.21.15.39:0/3763511173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4104070 msgr2=0x7f2ed40ff6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed37fe700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4104070 0x7f2ed40ff6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed37fe700 1 -- 172.21.15.39:0/3763511173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80095f0 con 0x7f2ed4101740 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed2ffd700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4104070 0x7f2ed40ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed37fe700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 0x7f2ed41010a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400d900 tx=0x7f2ec400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ec40041f0 con 0x7f2ed4101740 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed4100050 con 0x7f2ed4101740 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2ec4004ce0 con 0x7f2ed4101740 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ec400b750 con 0x7f2ed4101740 2024-02-19T19:10:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.456+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed419e4c0 con 0x7f2ed4101740 2024-02-19T19:10:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.458+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed4190620 con 0x7f2ed4101740 2024-02-19T19:10:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.458+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f2ec4004350 con 0x7f2ed4101740 2024-02-19T19:10:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.458+0000 7f2ed0ff9700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2ebc06d8a0 0x7f2ebc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:10:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.458+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2ec408d640 con 0x7f2ed4101740 2024-02-19T19:10:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.458+0000 7f2ed2ffd700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2ebc06d8a0 0x7f2ebc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:10:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.459+0000 7f2ed2ffd700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2ebc06d8a0 0x7f2ebc06fd60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8006010 tx=0x7f2ec800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:10:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.463+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ec405a970 con 0x7f2ed4101740 2024-02-19T19:10:37.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.761+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2ed402cc70 con 0x7f2ed4101740 2024-02-19T19:10:37.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.761+0000 7f2ed0ff9700 1 -- 172.21.15.39:0/3763511173 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f2ed402cc70 con 0x7f2ed4101740 2024-02-19T19:10:37.782 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:10:37.783 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:10:37.783 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2ebc06d8a0 msgr2=0x7f2ebc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2ebc06d8a0 0x7f2ebc06fd60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8006010 tx=0x7f2ec800b540 comp rx=0 tx=0).stop 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 msgr2=0x7f2ed41010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 0x7f2ed41010a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400d900 tx=0x7f2ec400dcc0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 shutdown_connections 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2ebc06d8a0 0x7f2ebc06fd60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ed4101740 0x7f2ed41010a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 --2- 172.21.15.39:0/3763511173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed4104070 0x7f2ed40ff6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 >> 172.21.15.39:0/3763511173 conn(0x7f2ed40fb150 msgr2=0x7f2ed40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 shutdown_connections 2024-02-19T19:10:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:10:37.763+0000 7f2ed9f1f700 1 -- 172.21.15.39:0/3763511173 wait complete. 2024-02-19T19:10:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:37 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/840202126' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:10:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:37 smithi039 ceph-mon[92224]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:38.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:38.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:37 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:38.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:37 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3763511173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:10:38.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:37 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/840202126' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:10:38.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:37 smithi086 ceph-mon[82675]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:10:38.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:38.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:37 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:38.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:37 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3763511173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:10:39.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:38 smithi039 ceph-mon[92224]: pgmap v168: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:39.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:38 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:39.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:38 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:39.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:38 smithi086 ceph-mon[82675]: pgmap v168: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:39.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:38 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:39.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:38 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:40.918 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:40 smithi039 ceph-mon[92224]: pgmap v169: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:41.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:40 smithi086 ceph-mon[82675]: pgmap v169: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:43.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: pgmap v170: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:43.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:42 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi039.aygwnv"]}]: dispatch 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: pgmap v170: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:43.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:10:43.287 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:42 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi039.aygwnv"]}]: dispatch 2024-02-19T19:10:44.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:43 smithi039 ceph-mon[92224]: Detected new or changed devices on smithi039 2024-02-19T19:10:44.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:43 smithi039 ceph-mon[92224]: Upgrade: It appears safe to stop mds.cephfs.smithi039.aygwnv 2024-02-19T19:10:44.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:44.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:43 smithi086 ceph-mon[82675]: Detected new or changed devices on smithi039 2024-02-19T19:10:44.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:43 smithi086 ceph-mon[82675]: Upgrade: It appears safe to stop mds.cephfs.smithi039.aygwnv 2024-02-19T19:10:44.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:45.150 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:44 smithi039 ceph-mon[92224]: pgmap v171: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:10:45.151 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:44 smithi086 ceph-mon[82675]: pgmap v171: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi039.aygwnv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:10:45.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:10:45.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:45 smithi039 ceph-mon[92224]: Upgrade: Updating mds.cephfs.smithi039.aygwnv 2024-02-19T19:10:45.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:45 smithi039 ceph-mon[92224]: Deploying daemon mds.cephfs.smithi039.aygwnv on smithi039 2024-02-19T19:10:46.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:45 smithi086 ceph-mon[82675]: Upgrade: Updating mds.cephfs.smithi039.aygwnv 2024-02-19T19:10:46.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:45 smithi086 ceph-mon[82675]: Deploying daemon mds.cephfs.smithi039.aygwnv on smithi039 2024-02-19T19:10:47.157 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:46 smithi039 ceph-mon[92224]: pgmap v172: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:47.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:46 smithi086 ceph-mon[82675]: pgmap v172: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:48.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:47 smithi039 ceph-mon[92224]: osdmap e83: 6 total, 6 up, 6 in 2024-02-19T19:10:48.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:47 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 2 up:standby 2024-02-19T19:10:48.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:47 smithi086 ceph-mon[82675]: osdmap e83: 6 total, 6 up, 6 in 2024-02-19T19:10:48.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:47 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 2 up:standby 2024-02-19T19:10:49.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:48 smithi039 ceph-mon[92224]: pgmap v174: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:49.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:48 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:10:49.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:48 smithi086 ceph-mon[82675]: pgmap v174: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:49.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:48 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:10:51.170 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:50 smithi039 ceph-mon[92224]: pgmap v175: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:51.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:50 smithi086 ceph-mon[82675]: pgmap v175: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: pgmap v176: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6828/3151633387,v1:172.21.15.39:6829/3151633387] up:boot 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:10:53.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:53 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:10:53.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: pgmap v176: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:53.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:53.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:53.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:10:53.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6828/3151633387,v1:172.21.15.39:6829/3151633387] up:boot 2024-02-19T19:10:53.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.euvbgw=up:active} 3 up:standby 2024-02-19T19:10:53.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:53 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:10:55.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:55 smithi039 ceph-mon[92224]: pgmap v177: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:55.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:55 smithi086 ceph-mon[82675]: pgmap v177: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:57.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:57 smithi039 ceph-mon[92224]: pgmap v178: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:57.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:57 smithi086 ceph-mon[82675]: pgmap v178: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:59.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:58 smithi039 ceph-mon[92224]: pgmap v179: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:59.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:59.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:59.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:10:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:10:59.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:58 smithi086 ceph-mon[82675]: pgmap v179: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:10:59.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:59.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:10:59.286 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:10:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:00.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:00 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:00.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:00 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:00.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:00 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:00.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:00 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:01.416 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:01 smithi039 ceph-mon[92224]: pgmap v180: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:01.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:01 smithi086 ceph-mon[82675]: pgmap v180: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:03.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:03 smithi039 ceph-mon[92224]: pgmap v181: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:03.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:03.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:03.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:03.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:11:03.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:03 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:03.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:03 smithi086 ceph-mon[82675]: pgmap v181: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:03.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:03.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:03.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:03.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:11:03.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:03 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi086.euvbgw"]}]: dispatch 2024-02-19T19:11:04.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:04 smithi039 ceph-mon[92224]: Upgrade: It appears safe to stop mds.cephfs.smithi086.euvbgw 2024-02-19T19:11:04.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:04.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi086.euvbgw"]}]: dispatch 2024-02-19T19:11:04.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:04 smithi086 ceph-mon[82675]: Upgrade: It appears safe to stop mds.cephfs.smithi086.euvbgw 2024-02-19T19:11:05.215 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:05 smithi086 ceph-mon[82675]: pgmap v182: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:05.215 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:05 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:05.215 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:05 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:11:05.215 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:05 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:05.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:05 smithi039 ceph-mon[92224]: pgmap v182: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:05.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:05 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:05.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:05 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.euvbgw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:11:05.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:05 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:06.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:06 smithi039 ceph-mon[92224]: Upgrade: Updating mds.cephfs.smithi086.euvbgw 2024-02-19T19:11:06.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:06 smithi039 ceph-mon[92224]: Deploying daemon mds.cephfs.smithi086.euvbgw on smithi086 2024-02-19T19:11:06.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:06 smithi086 ceph-mon[82675]: Upgrade: Updating mds.cephfs.smithi086.euvbgw 2024-02-19T19:11:06.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:06 smithi086 ceph-mon[82675]: Deploying daemon mds.cephfs.smithi086.euvbgw on smithi086 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:11:07.183+0000 7f0fee802700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: pgmap v183: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: osdmap e84: 6 total, 6 up, 6 in 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: Standby daemon mds.cephfs.smithi086.idtugy assigned to filesystem cephfs as rank 0 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:11:07.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:07 smithi039 ceph-mon[92224]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-02-19T19:11:07.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: pgmap v183: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:11:07.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:11:07.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:11:07.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: osdmap e84: 6 total, 6 up, 6 in 2024-02-19T19:11:07.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: Standby daemon mds.cephfs.smithi086.idtugy assigned to filesystem cephfs as rank 0 2024-02-19T19:11:07.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:11:07.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:07 smithi086 ceph-mon[82675]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-02-19T19:11:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 -- 172.21.15.39:0/118790287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe4100dc0 msgr2=0x7f9fe4101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:07.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 --2- 172.21.15.39:0/118790287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe4100dc0 0x7f9fe4101220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009910 tx=0x7f9fd0009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 -- 172.21.15.39:0/118790287 shutdown_connections 2024-02-19T19:11:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 --2- 172.21.15.39:0/118790287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe4100dc0 0x7f9fe4101220 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 --2- 172.21.15.39:0/118790287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe40ffbc0 0x7f9fe40fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 -- 172.21.15.39:0/118790287 >> 172.21.15.39:0/118790287 conn(0x7f9fe40fb160 msgr2=0x7f9fe40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.927+0000 7f9feb082700 1 -- 172.21.15.39:0/118790287 shutdown_connections 2024-02-19T19:11:07.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.928+0000 7f9feb082700 1 -- 172.21.15.39:0/118790287 wait complete. 2024-02-19T19:11:07.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.928+0000 7f9feb082700 1 Processor -- start 2024-02-19T19:11:07.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.928+0000 7f9feb082700 1 -- start start 2024-02-19T19:11:07.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9feb082700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe40ffbc0 0x7f9fe4196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9fe8e1e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe40ffbc0 0x7f9fe4196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9fe8e1e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe40ffbc0 0x7f9fe4196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42568/0 (socket says 172.21.15.39:42568) 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9fe8e1e700 1 -- 172.21.15.39:0/3229387966 learned_addr learned my addr 172.21.15.39:0/3229387966 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9feb082700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe4196f80 0x7f9fe419bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4197490 con 0x7f9fe40ffbc0 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4197600 con 0x7f9fe4196f80 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9fe3fff700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe4196f80 0x7f9fe419bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9fe3fff700 1 -- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe40ffbc0 msgr2=0x7f9fe4196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.929+0000 7f9fe3fff700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe40ffbc0 0x7f9fe4196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.930+0000 7f9fe3fff700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd00095f0 con 0x7f9fe4196f80 2024-02-19T19:11:07.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.930+0000 7f9fe3fff700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe4196f80 0x7f9fe419bff0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0008000 tx=0x7f9fd000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.930+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fd001d070 con 0x7f9fe4196f80 2024-02-19T19:11:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.930+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe419c530 con 0x7f9fe4196f80 2024-02-19T19:11:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.930+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9fd0005640 con 0x7f9fe4196f80 2024-02-19T19:11:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.930+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe419c9f0 con 0x7f9fe4196f80 2024-02-19T19:11:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.931+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fd00215e0 con 0x7f9fe4196f80 2024-02-19T19:11:07.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.931+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4068af0 con 0x7f9fe4196f80 2024-02-19T19:11:07.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.933+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f9fd0028080 con 0x7f9fe4196f80 2024-02-19T19:11:07.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.934+0000 7f9fe1ffb700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9fd406d910 0x7f9fd406fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:07.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.934+0000 7f9fe8e1e700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9fd406d910 0x7f9fd406fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:07.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.935+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9fd00572c0 con 0x7f9fe4196f80 2024-02-19T19:11:07.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.935+0000 7f9fe8e1e700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9fd406d910 0x7f9fd406fdd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8005950 tx=0x7f9fd8009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:07.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:07.942+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9fd005aa20 con 0x7f9fe4196f80 2024-02-19T19:11:08.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.387+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fe4105200 con 0x7f9fd406d910 2024-02-19T19:11:08.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.389+0000 7f9fe1ffb700 1 -- 172.21.15.39:0/3229387966 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9fe4105200 con 0x7f9fd406d910 2024-02-19T19:11:08.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9fd406d910 msgr2=0x7f9fd406fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:08.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9fd406d910 0x7f9fd406fdd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8005950 tx=0x7f9fd8009260 comp rx=0 tx=0).stop 2024-02-19T19:11:08.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe4196f80 msgr2=0x7f9fe419bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:08.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe4196f80 0x7f9fe419bff0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0008000 tx=0x7f9fd000fa00 comp rx=0 tx=0).stop 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 shutdown_connections 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f9fd406d910 0x7f9fd406fdd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fe40ffbc0 0x7f9fe4196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 --2- 172.21.15.39:0/3229387966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fe4196f80 0x7f9fe419bff0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 >> 172.21.15.39:0/3229387966 conn(0x7f9fe40fb160 msgr2=0x7f9fe4068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 shutdown_connections 2024-02-19T19:11:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.392+0000 7f9feb082700 1 -- 172.21.15.39:0/3229387966 wait complete. 2024-02-19T19:11:08.431 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:11:08.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:08 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi086.idtugy=up:replay} 2 up:standby 2024-02-19T19:11:08.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:08 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi086.idtugy=up:replay} 2 up:standby 2024-02-19T19:11:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 -- 172.21.15.39:0/2106657419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc0ffe70 msgr2=0x7f8adc1002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2106657419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc0ffe70 0x7f8adc1002d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009910 tx=0x7f8ac8009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 -- 172.21.15.39:0/2106657419 shutdown_connections 2024-02-19T19:11:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2106657419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc0ffe70 0x7f8adc1002d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2106657419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 -- 172.21.15.39:0/2106657419 >> 172.21.15.39:0/2106657419 conn(0x7f8adc0fa8c0 msgr2=0x7f8adc0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 -- 172.21.15.39:0/2106657419 shutdown_connections 2024-02-19T19:11:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.558+0000 7f8ae310f700 1 -- 172.21.15.39:0/2106657419 wait complete. 2024-02-19T19:11:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.559+0000 7f8ae310f700 1 Processor -- start 2024-02-19T19:11:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.559+0000 7f8ae310f700 1 -- start start 2024-02-19T19:11:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae310f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc0ff920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae310f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc1993c0 0x7f8adc19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae0eab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc0ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8adbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc1993c0 0x7f8adc19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae0eab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc0ff920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:33552/0 (socket says 172.21.15.39:33552) 2024-02-19T19:11:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae0eab700 1 -- 172.21.15.39:0/2704922314 learned_addr learned my addr 172.21.15.39:0/2704922314 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc19bdd0 con 0x7f8adc1993c0 2024-02-19T19:11:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc19bf10 con 0x7f8adc101130 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae0eab700 1 -- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc1993c0 msgr2=0x7f8adc19b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae0eab700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc1993c0 0x7f8adc19b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.560+0000 7f8ae0eab700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac80095f0 con 0x7f8adc101130 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.561+0000 7f8ae0eab700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc0ff920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000e8f0 tx=0x7f8ad000ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.561+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad0009790 con 0x7f8adc101130 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.561+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8ad000c810 con 0x7f8adc101130 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.561+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc19c1f0 con 0x7f8adc101130 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.561+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc19c740 con 0x7f8adc101130 2024-02-19T19:11:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.561+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad0003d20 con 0x7f8adc101130 2024-02-19T19:11:08.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.563+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc108710 con 0x7f8adc101130 2024-02-19T19:11:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.563+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f8ad0018430 con 0x7f8adc101130 2024-02-19T19:11:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.564+0000 7f8ad9ffb700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8acc06d8a0 0x7f8acc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.564+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8ad0014070 con 0x7f8adc101130 2024-02-19T19:11:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.564+0000 7f8adbfff700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8acc06d8a0 0x7f8acc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.565+0000 7f8adbfff700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8acc06d8a0 0x7f8acc06fd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8000c00 tx=0x7f8ac8005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:08.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:08.573+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8ad005ab80 con 0x7f8adc101130 2024-02-19T19:11:09.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.017+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8adc1057d0 con 0x7f8acc06d8a0 2024-02-19T19:11:09.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.019+0000 7f8ad9ffb700 1 -- 172.21.15.39:0/2704922314 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8adc1057d0 con 0x7f8acc06d8a0 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8acc06d8a0 msgr2=0x7f8acc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8acc06d8a0 0x7f8acc06fd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8000c00 tx=0x7f8ac8005a90 comp rx=0 tx=0).stop 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 msgr2=0x7f8adc0ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc0ff920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000e8f0 tx=0x7f8ad000ec00 comp rx=0 tx=0).stop 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 shutdown_connections 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8acc06d8a0 0x7f8acc06fd60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8adc101130 0x7f8adc0ff920 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 --2- 172.21.15.39:0/2704922314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8adc1993c0 0x7f8adc19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 >> 172.21.15.39:0/2704922314 conn(0x7f8adc0fa8c0 msgr2=0x7f8adc1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 shutdown_connections 2024-02-19T19:11:09.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.022+0000 7f8ae310f700 1 -- 172.21.15.39:0/2704922314 wait complete. 2024-02-19T19:11:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 -- 172.21.15.39:0/4089443736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60380ff320 msgr2=0x7f60380ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 --2- 172.21.15.39:0/4089443736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60380ff320 0x7f60380ff740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f602800aa20 tx=0x7f602800ad30 comp rx=0 tx=0).stop 2024-02-19T19:11:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 -- 172.21.15.39:0/4089443736 shutdown_connections 2024-02-19T19:11:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 --2- 172.21.15.39:0/4089443736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6038100520 0x7f6038100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 --2- 172.21.15.39:0/4089443736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60380ff320 0x7f60380ff740 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 -- 172.21.15.39:0/4089443736 >> 172.21.15.39:0/4089443736 conn(0x7f60380fa8c0 msgr2=0x7f60380fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 -- 172.21.15.39:0/4089443736 shutdown_connections 2024-02-19T19:11:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.133+0000 7f603e09a700 1 -- 172.21.15.39:0/4089443736 wait complete. 2024-02-19T19:11:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f603e09a700 1 Processor -- start 2024-02-19T19:11:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f603e09a700 1 -- start start 2024-02-19T19:11:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f603e09a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60380ff320 0x7f6038195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f603e09a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 0x7f60381940b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f603e09a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60381945f0 con 0x7f6038100520 2024-02-19T19:11:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f603e09a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6038194730 con 0x7f60380ff320 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.134+0000 7f60377fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60380ff320 0x7f6038195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 0x7f60381940b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 0x7f60381940b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42612/0 (socket says 172.21.15.39:42612) 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 -- 172.21.15.39:0/111346194 learned_addr learned my addr 172.21.15.39:0/111346194 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f60377fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60380ff320 0x7f6038195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:33564/0 (socket says 172.21.15.39:33564) 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 -- 172.21.15.39:0/111346194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60380ff320 msgr2=0x7f6038195a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60380ff320 0x7f6038195a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 -- 172.21.15.39:0/111346194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602800a700 con 0x7f6038100520 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.135+0000 7f6036ffd700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 0x7f60381940b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f602c00e920 tx=0x7f602c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.136+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f602c00c9d0 con 0x7f6038100520 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.136+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6038194a10 con 0x7f6038100520 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.136+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f602c010430 con 0x7f6038100520 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.136+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f602c018650 con 0x7f6038100520 2024-02-19T19:11:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.136+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6038194ea0 con 0x7f6038100520 2024-02-19T19:11:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.137+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60381075f0 con 0x7f6038100520 2024-02-19T19:11:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.138+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f602c0105a0 con 0x7f6038100520 2024-02-19T19:11:09.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.138+0000 7f6034ff9700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f602006d8a0 0x7f602006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.139+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f602c014070 con 0x7f6038100520 2024-02-19T19:11:09.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.139+0000 7f60377fe700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f602006d8a0 0x7f602006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.139+0000 7f60377fe700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f602006d8a0 0x7f602006fd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6028006010 tx=0x7f602800c650 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:09.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.147+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f602c05aa20 con 0x7f6038100520 2024-02-19T19:11:09.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.378+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6038104e70 con 0x7f602006d8a0 2024-02-19T19:11:09.401 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:09 smithi039 ceph-mon[92224]: pgmap v185: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2024-02-19T19:11:09.401 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:09 smithi039 ceph-mon[92224]: reconnect by client.14554 v1:172.21.15.86:0/3079196984 after 0 2024-02-19T19:11:09.401 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:09 smithi039 ceph-mon[92224]: reconnect by client.14548 v1:192.168.0.1:0/2680228978 after 0.000999978 2024-02-19T19:11:09.401 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:09 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:reconnect 2024-02-19T19:11:09.401 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:09 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi086.idtugy=up:reconnect} 2 up:standby 2024-02-19T19:11:09.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.398+0000 7f6034ff9700 1 -- 172.21.15.39:0/111346194 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6038104e70 con 0x7f602006d8a0 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (13m) 11s ago 14m 25.1M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (15m) 11s ago 15m 8724k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (13m) 54s ago 13m 8442k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (3m) 11s ago 15m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (3m) 54s ago 13m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (13m) 11s ago 14m 85.7M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (17s) 11s ago 8m 13.4M - 19.0.0-1450-gf78a58c0 f081f847070a 734d62a110c7 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (38s) 11s ago 9m 13.5M - 19.0.0-1450-gf78a58c0 f081f847070a 418296bb436e 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (9m) 54s ago 9m 29.2M - 18.2.1 94eadd990e0a 6afa612492a1 2024-02-19T19:11:09.421 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 running (8m) 54s ago 8m 17.9M - 18.2.1 94eadd990e0a af4a775bc97e 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (5m) 11s ago 16m 508M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (5m) 54s ago 13m 450M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (5m) 11s ago 16m 49.4M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (4m) 54s ago 13m 36.7M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (15m) 11s ago 15m 15.4M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (13m) 54s ago 13m 15.9M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (3m) 11s ago 12m 54.8M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (2m) 11s ago 12m 54.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 546fbde6ffab 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (2m) 11s ago 11m 44.2M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 3d2286828d68 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (2m) 54s ago 11m 44.9M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 37ce649da9a6 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (91s) 54s ago 10m 45.9M 2240M 19.0.0-1450-gf78a58c0 f081f847070a f6b44d6c5641 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (57s) 54s ago 10m 12.7M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 5b366510b403 2024-02-19T19:11:09.422 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (5m) 11s ago 14m 66.8M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f602006d8a0 msgr2=0x7f602006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f602006d8a0 0x7f602006fd60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6028006010 tx=0x7f602800c650 comp rx=0 tx=0).stop 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 msgr2=0x7f60381940b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 0x7f60381940b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f602c00e920 tx=0x7f602c00ec30 comp rx=0 tx=0).stop 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 shutdown_connections 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f602006d8a0 0x7f602006fd60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60380ff320 0x7f6038195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 --2- 172.21.15.39:0/111346194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6038100520 0x7f60381940b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 >> 172.21.15.39:0/111346194 conn(0x7f60380fa8c0 msgr2=0x7f6038103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:09.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 shutdown_connections 2024-02-19T19:11:09.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.401+0000 7f603e09a700 1 -- 172.21.15.39:0/111346194 wait complete. 2024-02-19T19:11:09.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 -- 172.21.15.39:0/1240266839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ff590 msgr2=0x7f1da40ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 --2- 172.21.15.39:0/1240266839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ff590 0x7f1da40ff9b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009910 tx=0x7f1d94009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:09.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 -- 172.21.15.39:0/1240266839 shutdown_connections 2024-02-19T19:11:09.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 --2- 172.21.15.39:0/1240266839 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da40ffef0 0x7f1da4100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 --2- 172.21.15.39:0/1240266839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ff590 0x7f1da40ff9b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 -- 172.21.15.39:0/1240266839 >> 172.21.15.39:0/1240266839 conn(0x7f1da40fb150 msgr2=0x7f1da40fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:09.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 -- 172.21.15.39:0/1240266839 shutdown_connections 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.511+0000 7f1da9694700 1 -- 172.21.15.39:0/1240266839 wait complete. 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.512+0000 7f1da9694700 1 Processor -- start 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.512+0000 7f1da9694700 1 -- start start 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.512+0000 7f1da9694700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da40ff590 0x7f1da41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.512+0000 7f1da9694700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 0x7f1da4198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.512+0000 7f1da9694700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da4199220 con 0x7f1da40ffef0 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.512+0000 7f1da9694700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da4199360 con 0x7f1da40ff590 2024-02-19T19:11:09.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da2ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da40ff590 0x7f1da41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da27fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 0x7f1da4198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da27fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 0x7f1da4198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42640/0 (socket says 172.21.15.39:42640) 2024-02-19T19:11:09.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da27fc700 1 -- 172.21.15.39:0/3725465544 learned_addr learned my addr 172.21.15.39:0/3725465544 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:09.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:09 smithi086 ceph-mon[82675]: pgmap v185: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2024-02-19T19:11:09.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:09 smithi086 ceph-mon[82675]: reconnect by client.14554 v1:172.21.15.86:0/3079196984 after 0 2024-02-19T19:11:09.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:09 smithi086 ceph-mon[82675]: reconnect by client.14548 v1:192.168.0.1:0/2680228978 after 0.000999978 2024-02-19T19:11:09.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:09 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:reconnect 2024-02-19T19:11:09.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:09 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi086.idtugy=up:reconnect} 2 up:standby 2024-02-19T19:11:09.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da27fc700 1 -- 172.21.15.39:0/3725465544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da40ff590 msgr2=0x7f1da41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da27fc700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da40ff590 0x7f1da41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.513+0000 7f1da27fc700 1 -- 172.21.15.39:0/3725465544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d940095f0 con 0x7f1da40ffef0 2024-02-19T19:11:09.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.514+0000 7f1da27fc700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 0x7f1da4198c00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800e920 tx=0x7f1d9800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:09.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.514+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d9800c9d0 con 0x7f1da40ffef0 2024-02-19T19:11:09.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.514+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1d98010430 con 0x7f1da40ffef0 2024-02-19T19:11:09.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.514+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da419de10 con 0x7f1da40ffef0 2024-02-19T19:11:09.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.514+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d980185f0 con 0x7f1da40ffef0 2024-02-19T19:11:09.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.514+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da419e330 con 0x7f1da40ffef0 2024-02-19T19:11:09.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.515+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da410ba10 con 0x7f1da40ffef0 2024-02-19T19:11:09.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.516+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f1d980105a0 con 0x7f1da40ffef0 2024-02-19T19:11:09.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.517+0000 7f1d8bfff700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1d8c06d8a0 0x7f1d8c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.517+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1d98014070 con 0x7f1da40ffef0 2024-02-19T19:11:09.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.517+0000 7f1da2ffd700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1d8c06d8a0 0x7f1d8c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.518+0000 7f1da2ffd700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1d8c06d8a0 0x7f1d8c06fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009fd0 tx=0x7f1d94005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:09.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.526+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1d9805a990 con 0x7f1da40ffef0 2024-02-19T19:11:09.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.822+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1da410bc00 con 0x7f1da40ffef0 2024-02-19T19:11:09.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.823+0000 7f1d8bfff700 1 -- 172.21.15.39:0/3725465544 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f1d9805a520 con 0x7f1da40ffef0 2024-02-19T19:11:09.845 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:11:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:11:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 12 2024-02-19T19:11:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:11:09.847 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1d8c06d8a0 msgr2=0x7f1d8c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1d8c06d8a0 0x7f1d8c06fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009fd0 tx=0x7f1d94005940 comp rx=0 tx=0).stop 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 msgr2=0x7f1da4198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 0x7f1da4198c00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800e920 tx=0x7f1d9800ec30 comp rx=0 tx=0).stop 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 shutdown_connections 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f1d8c06d8a0 0x7f1d8c06fd60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da40ff590 0x7f1da41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 --2- 172.21.15.39:0/3725465544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1da40ffef0 0x7f1da4198c00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 >> 172.21.15.39:0/3725465544 conn(0x7f1da40fb150 msgr2=0x7f1da4107bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 shutdown_connections 2024-02-19T19:11:09.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.825+0000 7f1da9694700 1 -- 172.21.15.39:0/3725465544 wait complete. 2024-02-19T19:11:09.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.942+0000 7f16a8c94700 1 -- 172.21.15.39:0/2459726728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a4101f40 msgr2=0x7f16a41023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.942+0000 7f16a8c94700 1 --2- 172.21.15.39:0/2459726728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a4101f40 0x7f16a41023a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1698009910 tx=0x7f1698009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:09.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.943+0000 7f16a8c94700 1 -- 172.21.15.39:0/2459726728 shutdown_connections 2024-02-19T19:11:09.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.943+0000 7f16a8c94700 1 --2- 172.21.15.39:0/2459726728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a4101f40 0x7f16a41023a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.943+0000 7f16a8c94700 1 --2- 172.21.15.39:0/2459726728 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a40fef60 0x7f16a40ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.943+0000 7f16a8c94700 1 -- 172.21.15.39:0/2459726728 >> 172.21.15.39:0/2459726728 conn(0x7f16a40fa8c0 msgr2=0x7f16a40fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.943+0000 7f16a8c94700 1 -- 172.21.15.39:0/2459726728 shutdown_connections 2024-02-19T19:11:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.943+0000 7f16a8c94700 1 -- 172.21.15.39:0/2459726728 wait complete. 2024-02-19T19:11:09.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a8c94700 1 Processor -- start 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a8c94700 1 -- start start 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a8c94700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a40fef60 0x7f16a41964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a8c94700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 0x7f16a4196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a8c94700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4197040 con 0x7f16a40fef60 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a8c94700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4197180 con 0x7f16a4101f40 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a40fef60 0x7f16a41964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a1d9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 0x7f16a4196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a1d9b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 0x7f16a4196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:33588/0 (socket says 172.21.15.39:33588) 2024-02-19T19:11:09.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a1d9b700 1 -- 172.21.15.39:0/3340107796 learned_addr learned my addr 172.21.15.39:0/3340107796 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a1d9b700 1 -- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a40fef60 msgr2=0x7f16a41964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.944+0000 7f16a1d9b700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a40fef60 0x7f16a41964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16a1d9b700 1 -- 172.21.15.39:0/3340107796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980095f0 con 0x7f16a4101f40 2024-02-19T19:11:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16a259c700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a40fef60 0x7f16a41964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:11:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16a1d9b700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 0x7f16a4196a20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1698000c00 tx=0x7f169800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:09.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f169801d070 con 0x7f16a4101f40 2024-02-19T19:11:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16a8c94700 1 -- 172.21.15.39:0/3340107796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a419bbd0 con 0x7f16a4101f40 2024-02-19T19:11:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1698005dc0 con 0x7f16a4101f40 2024-02-19T19:11:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16980226a0 con 0x7f16a4101f40 2024-02-19T19:11:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.945+0000 7f16a8c94700 1 -- 172.21.15.39:0/3340107796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a419c0c0 con 0x7f16a4101f40 2024-02-19T19:11:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.947+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16880052f0 con 0x7f16a4101f40 2024-02-19T19:11:09.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.947+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f16980038f0 con 0x7f16a4101f40 2024-02-19T19:11:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.947+0000 7f16937fe700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f168c06d8a0 0x7f168c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.948+0000 7f16a259c700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f168c06d8a0 0x7f168c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.948+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f169808e970 con 0x7f16a4101f40 2024-02-19T19:11:09.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.948+0000 7f16a259c700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f168c06d8a0 0x7f168c06fd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f16a4068380 tx=0x7f1694009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:09.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:09.956+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f169805bc70 con 0x7f16a4101f40 2024-02-19T19:11:10.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.218+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f16880061d0 con 0x7f16a4101f40 2024-02-19T19:11:10.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.218+0000 7f16937fe700 1 -- 172.21.15.39:0/3340107796 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1667 (secure 0 0 0) 0x7f1698003680 con 0x7f16a4101f40 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:e25 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 25 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:11:08.223616+0000 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:11:10.242 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 84 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24279} 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:11:10.243 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{0:24279} state up:rejoin seq 136 join_fscid=1 addr [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{-1:34380} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:11:10.244 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6828/3151633387,v1:172.21.15.39:6829/3151633387] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:11:10.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.221+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f168c06d8a0 msgr2=0x7f168c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.221+0000 7f16917fa700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f168c06d8a0 0x7f168c06fd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f16a4068380 tx=0x7f1694009260 comp rx=0 tx=0).stop 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.221+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 msgr2=0x7f16a4196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.221+0000 7f16917fa700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 0x7f16a4196a20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1698000c00 tx=0x7f169800f690 comp rx=0 tx=0).stop 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 shutdown_connections 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f168c06d8a0 0x7f168c06fd60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16a40fef60 0x7f16a41964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 --2- 172.21.15.39:0/3340107796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16a4101f40 0x7f16a4196a20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 >> 172.21.15.39:0/3340107796 conn(0x7f16a40fa8c0 msgr2=0x7f16a4105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 shutdown_connections 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.222+0000 7f16917fa700 1 -- 172.21.15.39:0/3340107796 wait complete. 2024-02-19T19:11:10.246 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 25 2024-02-19T19:11:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.380+0000 7f90c06f3700 1 -- 172.21.15.39:0/3987968992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b80fef80 msgr2=0x7f90b80ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.380+0000 7f90c06f3700 1 --2- 172.21.15.39:0/3987968992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b80fef80 0x7f90b80ff3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f90b4009910 tx=0x7f90b4009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 -- 172.21.15.39:0/3987968992 shutdown_connections 2024-02-19T19:11:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 --2- 172.21.15.39:0/3987968992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90b8102000 0x7f90b8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 --2- 172.21.15.39:0/3987968992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b80fef80 0x7f90b80ff3a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 -- 172.21.15.39:0/3987968992 >> 172.21.15.39:0/3987968992 conn(0x7f90b80fa8c0 msgr2=0x7f90b80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 -- 172.21.15.39:0/3987968992 shutdown_connections 2024-02-19T19:11:10.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 -- 172.21.15.39:0/3987968992 wait complete. 2024-02-19T19:11:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 Processor -- start 2024-02-19T19:11:10.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.381+0000 7f90c06f3700 1 -- start start 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90c06f3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90b80fef80 0x7f90b8196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90c06f3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 0x7f90b8196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90c06f3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b8197060 con 0x7f90b8102000 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90c06f3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b81971a0 con 0x7f90b80fef80 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90bdc8e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 0x7f90b8196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90bdc8e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 0x7f90b8196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42658/0 (socket says 172.21.15.39:42658) 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90bdc8e700 1 -- 172.21.15.39:0/175107109 learned_addr learned my addr 172.21.15.39:0/175107109 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90be48f700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90b80fef80 0x7f90b8196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90bdc8e700 1 -- 172.21.15.39:0/175107109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90b80fef80 msgr2=0x7f90b8196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90bdc8e700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90b80fef80 0x7f90b8196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.382+0000 7f90bdc8e700 1 -- 172.21.15.39:0/175107109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40095f0 con 0x7f90b8102000 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.383+0000 7f90bdc8e700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 0x7f90b8196a40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f90a800e920 tx=0x7f90a800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:10.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.383+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a8009790 con 0x7f90b8102000 2024-02-19T19:11:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.383+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b819bc50 con 0x7f90b8102000 2024-02-19T19:11:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.383+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b819c1a0 con 0x7f90b8102000 2024-02-19T19:11:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.383+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f90a800cba0 con 0x7f90b8102000 2024-02-19T19:11:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.384+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a8018630 con 0x7f90b8102000 2024-02-19T19:11:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.384+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b8109800 con 0x7f90b8102000 2024-02-19T19:11:10.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.385+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f90a800cd10 con 0x7f90b8102000 2024-02-19T19:11:10.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.385+0000 7f90af7fe700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f90a406d6a0 0x7f90a406fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:10.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.386+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f90a8014070 con 0x7f90b8102000 2024-02-19T19:11:10.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.386+0000 7f90be48f700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f90a406d6a0 0x7f90a406fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:10.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.386+0000 7f90be48f700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f90a406d6a0 0x7f90a406fb60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b760 tx=0x7f90b4009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:10.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.391+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f90a8052d20 con 0x7f90b8102000 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: from='client.44319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:rejoin 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi086.idtugy=up:rejoin} 2 up:standby 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: daemon mds.cephfs.smithi086.idtugy is now active in filesystem cephfs as rank 0 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3725465544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:10.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:10 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3340107796' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:11:10.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:10.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: from='client.44319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:10.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:rejoin 2024-02-19T19:11:10.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi086.idtugy=up:rejoin} 2 up:standby 2024-02-19T19:11:10.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: daemon mds.cephfs.smithi086.idtugy is now active in filesystem cephfs as rank 0 2024-02-19T19:11:10.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:10.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:11:10.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3725465544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:10.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:10 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3340107796' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:11:10.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.708+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90b819c8c0 con 0x7f90a406d6a0 2024-02-19T19:11:10.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.710+0000 7f90af7fe700 1 -- 172.21.15.39:0/175107109 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f90b819c8c0 con 0x7f90a406d6a0 2024-02-19T19:11:10.732 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:11:10.732 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:11:10.732 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:11:10.732 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "osd", 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "14/23 daemons upgraded", 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading mds daemons", 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:11:10.733 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:11:10.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f90a406d6a0 msgr2=0x7f90a406fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f90a406d6a0 0x7f90a406fb60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90b400b760 tx=0x7f90b4009f70 comp rx=0 tx=0).stop 2024-02-19T19:11:10.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 msgr2=0x7f90b8196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 0x7f90b8196a40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f90a800e920 tx=0x7f90a800ece0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 shutdown_connections 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f90a406d6a0 0x7f90a406fb60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90b80fef80 0x7f90b8196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 --2- 172.21.15.39:0/175107109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90b8102000 0x7f90b8196a40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 >> 172.21.15.39:0/175107109 conn(0x7f90b80fa8c0 msgr2=0x7f90b8105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 shutdown_connections 2024-02-19T19:11:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.713+0000 7f90c06f3700 1 -- 172.21.15.39:0/175107109 wait complete. 2024-02-19T19:11:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 -- 172.21.15.39:0/2328750039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 msgr2=0x7f8c701064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 --2- 172.21.15.39:0/2328750039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 0x7f8c701064c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009910 tx=0x7f8c5c009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 -- 172.21.15.39:0/2328750039 shutdown_connections 2024-02-19T19:11:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 --2- 172.21.15.39:0/2328750039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 0x7f8c701064c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 --2- 172.21.15.39:0/2328750039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 -- 172.21.15.39:0/2328750039 >> 172.21.15.39:0/2328750039 conn(0x7f8c700fb190 msgr2=0x7f8c700fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.850+0000 7f8c770b4700 1 -- 172.21.15.39:0/2328750039 shutdown_connections 2024-02-19T19:11:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.851+0000 7f8c770b4700 1 -- 172.21.15.39:0/2328750039 wait complete. 2024-02-19T19:11:10.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.851+0000 7f8c770b4700 1 Processor -- start 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.851+0000 7f8c770b4700 1 -- start start 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c770b4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c770b4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 0x7f8c70196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c770b4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70196fb0 con 0x7f8c701040d0 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c770b4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c701970f0 con 0x7f8c701017a0 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:33630/0 (socket says 172.21.15.39:33630) 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 -- 172.21.15.39:0/421360698 learned_addr learned my addr 172.21.15.39:0/421360698 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c6ffff700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 0x7f8c70196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 -- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 msgr2=0x7f8c70196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 0x7f8c70196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 -- 172.21.15.39:0/421360698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c0095f0 con 0x7f8c701017a0 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.852+0000 7f8c74e50700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70196450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400e920 tx=0x7f8c6400ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.853+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c6400c9d0 con 0x7f8c701017a0 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.853+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8c64010430 con 0x7f8c701017a0 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.853+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7019bba0 con 0x7f8c701017a0 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.853+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c64018650 con 0x7f8c701017a0 2024-02-19T19:11:10.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.853+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7019c0f0 con 0x7f8c701017a0 2024-02-19T19:11:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.854+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c70190610 con 0x7f8c701017a0 2024-02-19T19:11:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.855+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f8c640105a0 con 0x7f8c701017a0 2024-02-19T19:11:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.855+0000 7f8c6dffb700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8c6006d8a0 0x7f8c6006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.856+0000 7f8c6ffff700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8c6006d8a0 0x7f8c6006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.856+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8c64014070 con 0x7f8c701017a0 2024-02-19T19:11:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.856+0000 7f8c6ffff700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8c6006d8a0 0x7f8c6006fd60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009fd0 tx=0x7f8c5c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:10.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:10.864+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c64052d20 con 0x7f8c701017a0 2024-02-19T19:11:11.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.335+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8c7002cc70 con 0x7f8c701017a0 2024-02-19T19:11:11.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.335+0000 7f8c6dffb700 1 -- 172.21.15.39:0/421360698 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f8c6405ab10 con 0x7f8c701017a0 2024-02-19T19:11:11.358 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:11:11.358 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:11:11.358 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8c6006d8a0 msgr2=0x7f8c6006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8c6006d8a0 0x7f8c6006fd60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009fd0 tx=0x7f8c5c005940 comp rx=0 tx=0).stop 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 msgr2=0x7f8c70196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70196450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400e920 tx=0x7f8c6400ec30 comp rx=0 tx=0).stop 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 shutdown_connections 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8c6006d8a0 0x7f8c6006fd60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c701017a0 0x7f8c70196450 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 --2- 172.21.15.39:0/421360698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c701040d0 0x7f8c70196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 >> 172.21.15.39:0/421360698 conn(0x7f8c700fb190 msgr2=0x7f8c700fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 shutdown_connections 2024-02-19T19:11:11.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:11.338+0000 7f8c770b4700 1 -- 172.21.15.39:0/421360698 wait complete. 2024-02-19T19:11:11.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:11 smithi086 ceph-mon[82675]: pgmap v186: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1 op/s 2024-02-19T19:11:11.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:11 smithi086 ceph-mon[82675]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:11:11.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:11 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:active 2024-02-19T19:11:11.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:11 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.idtugy=up:active} 2 up:standby 2024-02-19T19:11:11.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:11 smithi039 ceph-mon[92224]: pgmap v186: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1 op/s 2024-02-19T19:11:11.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:11 smithi039 ceph-mon[92224]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:11:11.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:11 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.86:6826/978975893,v1:172.21.15.86:6827/978975893] up:active 2024-02-19T19:11:11.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:11 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.idtugy=up:active} 2 up:standby 2024-02-19T19:11:12.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:12 smithi086 ceph-mon[82675]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:12.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:12 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/421360698' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:11:12.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:12 smithi039 ceph-mon[92224]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:12.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:12 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/421360698' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:11:13.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:13 smithi086 ceph-mon[82675]: pgmap v187: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1 op/s 2024-02-19T19:11:13.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:13 smithi039 ceph-mon[92224]: pgmap v187: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1 op/s 2024-02-19T19:11:14.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:14 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:14.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:14 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:15.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:15 smithi039 ceph-mon[92224]: pgmap v188: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 7.0 KiB/s rd, 5 op/s 2024-02-19T19:11:15.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:15 smithi086 ceph-mon[82675]: pgmap v188: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 7.0 KiB/s rd, 5 op/s 2024-02-19T19:11:17.084 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to '1ce7ea8877c709fa862eac608550374bd96e166e'. 2024-02-19T19:11:17.084 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.084 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T19:11:17.084 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T19:11:17.084 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-02-19T19:11:17.085 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at 1ce7ea8877c fix 2024-02-19T19:11:17.090 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-19T19:11:17.111 INFO:tasks.workunit.client.0.smithi039.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T19:11:17.113 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T19:11:17.113 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T19:11:17.530 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:17 smithi086 ceph-mon[82675]: pgmap v189: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 9.4 KiB/s rd, 204 B/s wr, 8 op/s 2024-02-19T19:11:17.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:17 smithi039 ceph-mon[92224]: pgmap v189: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 9.4 KiB/s rd, 204 B/s wr, 8 op/s 2024-02-19T19:11:17.865 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T19:11:17.940 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T19:11:17.980 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T19:11:17.981 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T19:11:17.981 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T19:11:18.020 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T19:11:18.023 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T19:11:18.023 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-19T19:11:18.078 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-19T19:11:18.080 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-19T19:11:18.080 DEBUG:teuthology.orchestra.run.smithi039:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-19T19:11:18.138 INFO:tasks.workunit.client.0.smithi039.stderr:+ mkdir -p fsstress 2024-02-19T19:11:18.140 INFO:tasks.workunit.client.0.smithi039.stderr:+ pushd fsstress 2024-02-19T19:11:18.140 INFO:tasks.workunit.client.0.smithi039.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-19T19:11:18.141 INFO:tasks.workunit.client.0.smithi039.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-19T19:11:19.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:19 smithi039 ceph-mon[92224]: pgmap v190: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 198 B/s wr, 8 op/s 2024-02-19T19:11:19.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:19 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:19.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:19 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:19.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:19 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:19.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:19 smithi086 ceph-mon[82675]: pgmap v190: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 9.1 KiB/s rd, 198 B/s wr, 8 op/s 2024-02-19T19:11:19.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:19 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:19.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:19 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:19.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:19 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:20 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.86:6824/3342991439,v1:172.21.15.86:6825/3342991439] up:boot 2024-02-19T19:11:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:20 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi086.idtugy=up:active} 3 up:standby 2024-02-19T19:11:20.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:20 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:11:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:20 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.86:6824/3342991439,v1:172.21.15.86:6825/3342991439] up:boot 2024-02-19T19:11:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:20 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi086.idtugy=up:active} 3 up:standby 2024-02-19T19:11:20.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:20 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:11:21.387 INFO:tasks.workunit.client.0.smithi039.stderr:+ tar xzf ltp-full.tgz 2024-02-19T19:11:21.669 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:21 smithi086 ceph-mon[82675]: pgmap v191: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 7.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-02-19T19:11:21.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:21 smithi039 ceph-mon[92224]: pgmap v191: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 7.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-02-19T19:11:23.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:23 smithi039 ceph-mon[92224]: pgmap v192: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-02-19T19:11:23.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:23.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:23 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:23.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:23 smithi086 ceph-mon[82675]: pgmap v192: 65 pgs: 65 active+clean; 591 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-02-19T19:11:23.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:23 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:23.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:23 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:25.455 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:25 smithi086 ceph-mon[82675]: pgmap v193: 65 pgs: 65 active+clean; 591 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-02-19T19:11:25.455 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:25 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:25.455 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:25 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:25.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:25 smithi039 ceph-mon[92224]: pgmap v193: 65 pgs: 65 active+clean; 591 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 5.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-02-19T19:11:25.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:25.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:25 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:27.231 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:27 smithi086 ceph-mon[82675]: pgmap v194: 65 pgs: 65 active+clean; 4.7 MiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 370 KiB/s wr, 26 op/s 2024-02-19T19:11:27.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:27 smithi039 ceph-mon[92224]: pgmap v194: 65 pgs: 65 active+clean; 4.7 MiB data, 184 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 370 KiB/s wr, 26 op/s 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: pgmap v195: 65 pgs: 65 active+clean; 16 MiB data, 227 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s wr, 106 op/s 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:11:29.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:29 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:29.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: pgmap v195: 65 pgs: 65 active+clean; 16 MiB data, 227 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s wr, 106 op/s 2024-02-19T19:11:29.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:29.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:29.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:29.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:29.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:11:29.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:29 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:30.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: Detected new or changed devices on smithi086 2024-02-19T19:11:30.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:30.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.495 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi086.idtugy"]}]: dispatch 2024-02-19T19:11:30.496 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:30 smithi086 ceph-mon[82675]: Upgrade: It appears safe to stop mds.cephfs.smithi086.idtugy 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: Detected new or changed devices on smithi086 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi086.idtugy"]}]: dispatch 2024-02-19T19:11:30.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:30 smithi039 ceph-mon[92224]: Upgrade: It appears safe to stop mds.cephfs.smithi086.idtugy 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr:Note: switching to '1ce7ea8877c709fa862eac608550374bd96e166e'. 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr: git switch -c 2024-02-19T19:11:31.333 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr:Or undo this operation with: 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr: git switch - 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr: 2024-02-19T19:11:31.334 INFO:tasks.workunit.client.1.smithi086.stderr:HEAD is now at 1ce7ea8877c fix 2024-02-19T19:11:31.338 DEBUG:teuthology.orchestra.run.smithi086:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-19T19:11:31.390 INFO:tasks.workunit.client.1.smithi086.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T19:11:31.393 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-19T19:11:31.393 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T19:11:31.429 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:31 smithi086 ceph-mon[82675]: pgmap v196: 65 pgs: 65 active+clean; 29 MiB data, 276 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s wr, 152 op/s 2024-02-19T19:11:31.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:31 smithi039 ceph-mon[92224]: pgmap v196: 65 pgs: 65 active+clean; 29 MiB data, 276 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s wr, 152 op/s 2024-02-19T19:11:32.532 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T19:11:32.646 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T19:11:32.704 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-19T19:11:32.706 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-19T19:11:32.706 INFO:tasks.workunit.client.1.smithi086.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T19:11:32.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:32 smithi039 ceph-mon[92224]: Upgrade: Updating mds.cephfs.smithi086.idtugy 2024-02-19T19:11:32.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:32.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:11:32.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:32 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:32.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:32 smithi039 ceph-mon[92224]: Deploying daemon mds.cephfs.smithi086.idtugy on smithi086 2024-02-19T19:11:32.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:32 smithi039 ceph-mon[92224]: pgmap v197: 65 pgs: 65 active+clean; 29 MiB data, 281 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s wr, 152 op/s 2024-02-19T19:11:32.750 INFO:tasks.workunit.client.1.smithi086.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-19T19:11:32.753 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T19:11:32.753 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-19T19:11:32.768 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-19T19:11:32.769 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-19T19:11:32.769 DEBUG:teuthology.orchestra.run.smithi086:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-19T19:11:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:32 smithi086 ceph-mon[82675]: Upgrade: Updating mds.cephfs.smithi086.idtugy 2024-02-19T19:11:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi086.idtugy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:11:32.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:32 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:32.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:32 smithi086 ceph-mon[82675]: Deploying daemon mds.cephfs.smithi086.idtugy on smithi086 2024-02-19T19:11:32.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:32 smithi086 ceph-mon[82675]: pgmap v197: 65 pgs: 65 active+clean; 29 MiB data, 281 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s wr, 152 op/s 2024-02-19T19:11:32.808 INFO:tasks.workunit.client.1.smithi086.stderr:+ mkdir -p fsstress 2024-02-19T19:11:32.810 INFO:tasks.workunit.client.1.smithi086.stderr:+ pushd fsstress 2024-02-19T19:11:32.810 INFO:tasks.workunit.client.1.smithi086.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-19T19:11:32.810 INFO:tasks.workunit.client.1.smithi086.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-19T19:11:34.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:11:33.833+0000 7f0fee802700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: pgmap v198: 65 pgs: 65 active+clean; 33 MiB data, 295 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s wr, 152 op/s 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: osdmap e85: 6 total, 6 up, 6 in 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: Standby daemon mds.cephfs.smithi039.omsmsr assigned to filesystem cephfs as rank 0 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:replay} 2 up:standby 2024-02-19T19:11:34.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:34 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: pgmap v198: 65 pgs: 65 active+clean; 33 MiB data, 295 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s wr, 152 op/s 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: osdmap e85: 6 total, 6 up, 6 in 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: Standby daemon mds.cephfs.smithi039.omsmsr assigned to filesystem cephfs as rank 0 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:replay} 2 up:standby 2024-02-19T19:11:35.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:34 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:11:36.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:36 smithi039 ceph-mon[92224]: pgmap v200: 65 pgs: 65 active+clean; 37 MiB data, 303 MiB used, 536 GiB / 536 GiB avail; 469 KiB/s rd, 3.4 MiB/s wr, 155 op/s 2024-02-19T19:11:36.834 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:36 smithi039 ceph-mon[92224]: reconnect by client.14554 v1:172.21.15.86:0/3079196984 after 0 2024-02-19T19:11:36.834 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:36 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:reconnect 2024-02-19T19:11:36.834 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:36 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:reconnect} 2 up:standby 2024-02-19T19:11:36.834 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:36 smithi039 ceph-mon[92224]: reconnect by client.14548 v1:192.168.0.1:0/2680228978 after 0.0209996 2024-02-19T19:11:37.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:36 smithi086 ceph-mon[82675]: pgmap v200: 65 pgs: 65 active+clean; 37 MiB data, 303 MiB used, 536 GiB / 536 GiB avail; 469 KiB/s rd, 3.4 MiB/s wr, 155 op/s 2024-02-19T19:11:37.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:36 smithi086 ceph-mon[82675]: reconnect by client.14554 v1:172.21.15.86:0/3079196984 after 0 2024-02-19T19:11:37.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:36 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:reconnect 2024-02-19T19:11:37.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:36 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:reconnect} 2 up:standby 2024-02-19T19:11:37.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:36 smithi086 ceph-mon[82675]: reconnect by client.14548 v1:192.168.0.1:0/2680228978 after 0.0209996 2024-02-19T19:11:37.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:37 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:rejoin 2024-02-19T19:11:37.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:37 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:rejoin} 2 up:standby 2024-02-19T19:11:38.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:37 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:rejoin 2024-02-19T19:11:38.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:37 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:rejoin} 2 up:standby 2024-02-19T19:11:38.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:38 smithi086 ceph-mon[82675]: pgmap v201: 65 pgs: 65 active+clean; 41 MiB data, 320 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-02-19T19:11:38.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:38 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:clientreplay 2024-02-19T19:11:38.887 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:38 smithi086 ceph-mon[82675]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:clientreplay} 2 up:standby 2024-02-19T19:11:38.888 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:38 smithi086 ceph-mon[82675]: daemon mds.cephfs.smithi039.omsmsr is now active in filesystem cephfs as rank 0 2024-02-19T19:11:38.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:38 smithi039 ceph-mon[92224]: pgmap v201: 65 pgs: 65 active+clean; 41 MiB data, 320 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 62 op/s 2024-02-19T19:11:38.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:38 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:clientreplay 2024-02-19T19:11:38.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:38 smithi039 ceph-mon[92224]: fsmap cephfs:1/1 {0=cephfs.smithi039.omsmsr=up:clientreplay} 2 up:standby 2024-02-19T19:11:38.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:38 smithi039 ceph-mon[92224]: daemon mds.cephfs.smithi039.omsmsr is now active in filesystem cephfs as rank 0 2024-02-19T19:11:39.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:39 smithi039 ceph-mon[92224]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:11:39.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:39 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:active 2024-02-19T19:11:39.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:39 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 2 up:standby 2024-02-19T19:11:40.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:39 smithi086 ceph-mon[82675]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:11:40.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:39 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] up:active 2024-02-19T19:11:40.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:39 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 2 up:standby 2024-02-19T19:11:40.638 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:40 smithi086 ceph-mon[82675]: pgmap v202: 65 pgs: 65 active+clean; 49 MiB data, 328 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 85 op/s 2024-02-19T19:11:40.942 INFO:tasks.workunit.client.1.smithi086.stderr:+ tar xzf ltp-full.tgz 2024-02-19T19:11:40.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:40 smithi039 ceph-mon[92224]: pgmap v202: 65 pgs: 65 active+clean; 49 MiB data, 328 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 85 op/s 2024-02-19T19:11:41.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.545+0000 7fc578d49700 1 -- 172.21.15.39:0/938334020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc57410a700 msgr2=0x7fc57410cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:41.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.545+0000 7fc578d49700 1 --2- 172.21.15.39:0/938334020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc57410a700 0x7fc57410cb90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc568009b80 comp rx=0 tx=0).stop 2024-02-19T19:11:41.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.546+0000 7fc578d49700 1 -- 172.21.15.39:0/938334020 shutdown_connections 2024-02-19T19:11:41.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.546+0000 7fc578d49700 1 --2- 172.21.15.39:0/938334020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc57410a700 0x7fc57410cb90 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc568009870 tx=0x7fc568009b80 comp rx=0 tx=0).stop 2024-02-19T19:11:41.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.546+0000 7fc578d49700 1 --2- 172.21.15.39:0/938334020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc574107d90 0x7fc57410a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:41.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.546+0000 7fc578d49700 1 -- 172.21.15.39:0/938334020 >> 172.21.15.39:0/938334020 conn(0x7fc57406da40 msgr2=0x7fc57406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:41.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.549+0000 7fc578d49700 1 -- 172.21.15.39:0/938334020 shutdown_connections 2024-02-19T19:11:41.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.550+0000 7fc578d49700 1 -- 172.21.15.39:0/938334020 wait complete. 2024-02-19T19:11:41.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.551+0000 7fc578d49700 1 Processor -- start 2024-02-19T19:11:41.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.551+0000 7fc578d49700 1 -- start start 2024-02-19T19:11:41.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.553+0000 7fc578d49700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 0x7fc5741a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:41.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.553+0000 7fc578d49700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5741a59c0 0x7fc5741aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:41.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.553+0000 7fc578d49700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5741a5ed0 con 0x7fc574107d90 2024-02-19T19:11:41.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.553+0000 7fc578d49700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5741a6040 con 0x7fc5741a59c0 2024-02-19T19:11:41.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.557+0000 7fc57259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 0x7fc5741a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:41.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.557+0000 7fc57259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 0x7fc5741a5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59410/0 (socket says 172.21.15.39:59410) 2024-02-19T19:11:41.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.557+0000 7fc57259c700 1 -- 172.21.15.39:0/4193192751 learned_addr learned my addr 172.21.15.39:0/4193192751 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:41.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.559+0000 7fc57259c700 1 -- 172.21.15.39:0/4193192751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5741a59c0 msgr2=0x7fc5741aaa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T19:11:41.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.559+0000 7fc57259c700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5741a59c0 0x7fc5741aaa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:41.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.559+0000 7fc57259c700 1 -- 172.21.15.39:0/4193192751 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc568009520 con 0x7fc574107d90 2024-02-19T19:11:41.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.561+0000 7fc57259c700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 0x7fc5741a5480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e940 tx=0x7fc56400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:41.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.563+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc56400eee0 con 0x7fc574107d90 2024-02-19T19:11:41.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.564+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc564004ce0 con 0x7fc574107d90 2024-02-19T19:11:41.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.564+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc564005650 con 0x7fc574107d90 2024-02-19T19:11:41.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.564+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5741aafd0 con 0x7fc574107d90 2024-02-19T19:11:41.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.564+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5741ab470 con 0x7fc574107d90 2024-02-19T19:11:41.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.566+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc554005320 con 0x7fc574107d90 2024-02-19T19:11:41.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.570+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fc564004750 con 0x7fc574107d90 2024-02-19T19:11:41.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.571+0000 7fc5637fe700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc55c06da10 0x7fc55c06fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:41.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.571+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc564014070 con 0x7fc574107d90 2024-02-19T19:11:41.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.572+0000 7fc571d9b700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc55c06da10 0x7fc55c06fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:41.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.573+0000 7fc571d9b700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc55c06da10 0x7fc55c06fed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc568009f50 tx=0x7fc56800b4e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:41.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:41.579+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc56405ab50 con 0x7fc574107d90 2024-02-19T19:11:42.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.042+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc554000bf0 con 0x7fc55c06da10 2024-02-19T19:11:42.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.044+0000 7fc5637fe700 1 -- 172.21.15.39:0/4193192751 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc554000bf0 con 0x7fc55c06da10 2024-02-19T19:11:42.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.051+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc55c06da10 msgr2=0x7fc55c06fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.051+0000 7fc578d49700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc55c06da10 0x7fc55c06fed0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc568009f50 tx=0x7fc56800b4e0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.051+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 msgr2=0x7fc5741a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.051+0000 7fc578d49700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 0x7fc5741a5480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc56400e940 tx=0x7fc56400ed00 comp rx=0 tx=0).stop 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 shutdown_connections 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc55c06da10 0x7fc55c06fed0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc574107d90 0x7fc5741a5480 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 --2- 172.21.15.39:0/4193192751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5741a59c0 0x7fc5741aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 >> 172.21.15.39:0/4193192751 conn(0x7fc57406da40 msgr2=0x7fc57406fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 shutdown_connections 2024-02-19T19:11:42.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.053+0000 7fc578d49700 1 -- 172.21.15.39:0/4193192751 wait complete. 2024-02-19T19:11:42.095 INFO:teuthology.orchestra.run.smithi039.stdout:true 2024-02-19T19:11:42.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.248+0000 7fbd10999700 1 -- 172.21.15.39:0/461942111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 msgr2=0x7fbd0c10caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.248+0000 7fbd10999700 1 --2- 172.21.15.39:0/461942111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 0x7fbd0c10caf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400b0a0 tx=0x7fbd0400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.252+0000 7fbd10999700 1 -- 172.21.15.39:0/461942111 shutdown_connections 2024-02-19T19:11:42.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.252+0000 7fbd10999700 1 --2- 172.21.15.39:0/461942111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 0x7fbd0c10caf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.252+0000 7fbd10999700 1 --2- 172.21.15.39:0/461942111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c10a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.252+0000 7fbd10999700 1 -- 172.21.15.39:0/461942111 >> 172.21.15.39:0/461942111 conn(0x7fbd0c06dcd0 msgr2=0x7fbd0c0700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:42.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.254+0000 7fbd10999700 1 -- 172.21.15.39:0/461942111 shutdown_connections 2024-02-19T19:11:42.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.254+0000 7fbd10999700 1 -- 172.21.15.39:0/461942111 wait complete. 2024-02-19T19:11:42.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.255+0000 7fbd10999700 1 Processor -- start 2024-02-19T19:11:42.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.255+0000 7fbd10999700 1 -- start start 2024-02-19T19:11:42.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd10999700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c1a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:42.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd10999700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 0x7fbd0c1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:42.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd10999700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1a5c00 con 0x7fbd0c107d90 2024-02-19T19:11:42.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd10999700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c1a5d40 con 0x7fbd0c10a6c0 2024-02-19T19:11:42.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c1a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:42.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c1a50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59426/0 (socket says 172.21.15.39:59426) 2024-02-19T19:11:42.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.256+0000 7fbd0b7fe700 1 -- 172.21.15.39:0/1802989041 learned_addr learned my addr 172.21.15.39:0/1802989041 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:42.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.258+0000 7fbd0affd700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 0x7fbd0c1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:42.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.258+0000 7fbd0b7fe700 1 -- 172.21.15.39:0/1802989041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 msgr2=0x7fbd0c1a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.258+0000 7fbd0b7fe700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 0x7fbd0c1a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.258+0000 7fbd0b7fe700 1 -- 172.21.15.39:0/1802989041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04009d00 con 0x7fbd0c107d90 2024-02-19T19:11:42.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.259+0000 7fbd0b7fe700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c1a50a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009a40 tx=0x7fbcfc00ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:42.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.260+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcfc00cc60 con 0x7fbd0c107d90 2024-02-19T19:11:42.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.260+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c1aa7f0 con 0x7fbd0c107d90 2024-02-19T19:11:42.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.261+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c1aadc0 con 0x7fbd0c107d90 2024-02-19T19:11:42.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.261+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbcfc00cdc0 con 0x7fbd0c107d90 2024-02-19T19:11:42.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.261+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcfc0163f0 con 0x7fbd0c107d90 2024-02-19T19:11:42.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.263+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fbcfc004750 con 0x7fbd0c107d90 2024-02-19T19:11:42.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.264+0000 7fbd08ff9700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbcf406d9c0 0x7fbcf406fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:42.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.265+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fbcfc08cff0 con 0x7fbd0c107d90 2024-02-19T19:11:42.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.265+0000 7fbd0affd700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbcf406d9c0 0x7fbcf406fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:42.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.266+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf8005320 con 0x7fbd0c107d90 2024-02-19T19:11:42.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.269+0000 7fbd0affd700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbcf406d9c0 0x7fbcf406fe80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009fd0 tx=0x7fbd0400b720 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:42.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.276+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbcfc05a300 con 0x7fbd0c107d90 2024-02-19T19:11:42.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.743+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcf8000bf0 con 0x7fbcf406d9c0 2024-02-19T19:11:42.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.746+0000 7fbd08ff9700 1 -- 172.21.15.39:0/1802989041 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbcf8000bf0 con 0x7fbcf406d9c0 2024-02-19T19:11:42.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.750+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbcf406d9c0 msgr2=0x7fbcf406fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.750+0000 7fbd10999700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbcf406d9c0 0x7fbcf406fe80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009fd0 tx=0x7fbd0400b720 comp rx=0 tx=0).stop 2024-02-19T19:11:42.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.750+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 msgr2=0x7fbd0c1a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.750+0000 7fbd10999700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c1a50a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009a40 tx=0x7fbcfc00ede0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.752+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 shutdown_connections 2024-02-19T19:11:42.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.752+0000 7fbd10999700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fbcf406d9c0 0x7fbcf406fe80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.752+0000 7fbd10999700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c107d90 0x7fbd0c1a50a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.752+0000 7fbd10999700 1 --2- 172.21.15.39:0/1802989041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd0c10a6c0 0x7fbd0c1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.752+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 >> 172.21.15.39:0/1802989041 conn(0x7fbd0c06dcd0 msgr2=0x7fbd0c0700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:42.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.753+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 shutdown_connections 2024-02-19T19:11:42.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.753+0000 7fbd10999700 1 -- 172.21.15.39:0/1802989041 wait complete. 2024-02-19T19:11:42.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.953+0000 7f7f119bf700 1 -- 172.21.15.39:0/981823085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 msgr2=0x7f7f04094610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.953+0000 7f7f119bf700 1 --2- 172.21.15.39:0/981823085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f04094610 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8009910 tx=0x7f7ef8009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:42.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.955+0000 7f7f119bf700 1 -- 172.21.15.39:0/981823085 shutdown_connections 2024-02-19T19:11:42.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.955+0000 7f7f119bf700 1 --2- 172.21.15.39:0/981823085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f04095850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.955+0000 7f7f119bf700 1 --2- 172.21.15.39:0/981823085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f04094610 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.955+0000 7f7f119bf700 1 -- 172.21.15.39:0/981823085 >> 172.21.15.39:0/981823085 conn(0x7f7f0408f770 msgr2=0x7f7f04091bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:42.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.955+0000 7f7f119bf700 1 -- 172.21.15.39:0/981823085 shutdown_connections 2024-02-19T19:11:42.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.955+0000 7f7f119bf700 1 -- 172.21.15.39:0/981823085 wait complete. 2024-02-19T19:11:42.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.956+0000 7f7f119bf700 1 Processor -- start 2024-02-19T19:11:42.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.958+0000 7f7f119bf700 1 -- start start 2024-02-19T19:11:42.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.958+0000 7f7f119bf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f040a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:42.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.958+0000 7f7f119bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f040a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.958+0000 7f7f119bf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040a4f50 con 0x7f7f040941f0 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.958+0000 7f7f119bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f040a5090 con 0x7f7f040953f0 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.959+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f040a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.959+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f040a49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:50904/0 (socket says 172.21.15.39:50904) 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.959+0000 7f7f0bfff700 1 -- 172.21.15.39:0/1918932859 learned_addr learned my addr 172.21.15.39:0/1918932859 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.960+0000 7f7f0bfff700 1 -- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 msgr2=0x7f7f040a4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:42.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.960+0000 7f7f109bd700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f040a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.960+0000 7f7f0bfff700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f040a4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.960+0000 7f7f0bfff700 1 -- 172.21.15.39:0/1918932859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef80095f0 con 0x7f7f040953f0 2024-02-19T19:11:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.960+0000 7f7f109bd700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f040a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:11:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.961+0000 7f7f0bfff700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f040a49c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0665f0 tx=0x7f7f0c069d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.962+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0c069570 con 0x7f7f040953f0 2024-02-19T19:11:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.962+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7f0c075040 con 0x7f7f040953f0 2024-02-19T19:11:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.962+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f040a9b50 con 0x7f7f040953f0 2024-02-19T19:11:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.962+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0c07c3d0 con 0x7f7f040953f0 2024-02-19T19:11:42.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.962+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f040aa070 con 0x7f7f040953f0 2024-02-19T19:11:42.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.964+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f040aa320 con 0x7f7f040953f0 2024-02-19T19:11:42.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.966+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f7f0c067710 con 0x7f7f040953f0 2024-02-19T19:11:42.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.966+0000 7f7f09ffb700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7efc06d940 0x7f7efc06fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:42.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.967+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7f0c0f8520 con 0x7f7f040953f0 2024-02-19T19:11:42.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.969+0000 7f7f109bd700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7efc06d940 0x7f7efc06fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:42.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.970+0000 7f7f109bd700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7efc06d940 0x7f7efc06fe00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8006010 tx=0x7f7ef8005dc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:42.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:42.972+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7f0c0c5830 con 0x7f7f040953f0 2024-02-19T19:11:43.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.271+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f04140290 con 0x7f7efc06d940 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.288+0000 7f7f09ffb700 1 -- 172.21.15.39:0/1918932859 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7f04140290 con 0x7f7efc06d940 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stdout:alertmanager.smithi039 smithi039 *:9093,9094 running (13m) 45s ago 15m 25.1M - 0.25.0 c8568f914cd2 b1fcecaac900 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi039 smithi039 running (15m) 45s ago 15m 8724k - 18.2.1 94eadd990e0a cdfd2eeb0ad2 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-exporter.smithi086 smithi086 running (14m) 20s ago 14m 8720k - 18.2.1 94eadd990e0a 3ad8b6c1f8d2 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi039 smithi039 running (4m) 45s ago 15m 6669k - 19.0.0-1450-gf78a58c0 f081f847070a ba0f2288d0af 2024-02-19T19:11:43.311 INFO:teuthology.orchestra.run.smithi039.stdout:crash.smithi086 smithi086 running (4m) 20s ago 14m 6665k - 19.0.0-1450-gf78a58c0 f081f847070a 1c6a2621732b 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:grafana.smithi039 smithi039 *:3000 running (13m) 45s ago 15m 85.7M - 9.4.7 954c08fa6188 ee954ddbd73a 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.aygwnv smithi039 running (51s) 45s ago 9m 13.4M - 19.0.0-1450-gf78a58c0 f081f847070a 734d62a110c7 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi039.omsmsr smithi039 running (71s) 45s ago 9m 13.5M - 19.0.0-1450-gf78a58c0 f081f847070a 418296bb436e 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.euvbgw smithi086 running (24s) 20s ago 9m 13.9M - 19.0.0-1450-gf78a58c0 f081f847070a c09b00143666 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mds.cephfs.smithi086.idtugy smithi086 starting - - - - 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi039.ydmpuh smithi039 *:8443,9283,8765 running (6m) 45s ago 16m 508M - 19.0.0-1450-gf78a58c0 f081f847070a 3d4315692b57 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.smithi086.huubbu smithi086 *:8443,9283,8765 running (5m) 20s ago 14m 451M - 19.0.0-1450-gf78a58c0 f081f847070a fbd829f78f68 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi039 smithi039 running (5m) 45s ago 16m 49.4M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 3714643bde4a 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:mon.smithi086 smithi086 running (5m) 20s ago 14m 40.4M 2048M 19.0.0-1450-gf78a58c0 f081f847070a 7dc732d9913b 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi039 smithi039 *:9100 running (15m) 45s ago 15m 15.4M - 1.5.0 0da6a335fe13 39f31e9a9a44 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:node-exporter.smithi086 smithi086 *:9100 running (14m) 20s ago 14m 16.1M - 1.5.0 0da6a335fe13 19bceea5d35c 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (4m) 45s ago 13m 54.8M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 601c2901cc7d 2024-02-19T19:11:43.312 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi039 running (3m) 45s ago 12m 54.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 546fbde6ffab 2024-02-19T19:11:43.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi039 running (3m) 45s ago 12m 44.2M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 3d2286828d68 2024-02-19T19:11:43.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi086 running (2m) 20s ago 11m 46.2M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 37ce649da9a6 2024-02-19T19:11:43.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi086 running (2m) 20s ago 11m 46.0M 2240M 19.0.0-1450-gf78a58c0 f081f847070a f6b44d6c5641 2024-02-19T19:11:43.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi086 running (91s) 20s ago 11m 44.2M 2240M 19.0.0-1450-gf78a58c0 f081f847070a 5b366510b403 2024-02-19T19:11:43.313 INFO:teuthology.orchestra.run.smithi039.stdout:prometheus.smithi039 smithi039 *:9095 running (5m) 45s ago 14m 66.8M - 2.43.0 a07b618ecd1d 15c825eb7ea9 2024-02-19T19:11:43.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7efc06d940 msgr2=0x7f7efc06fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7efc06d940 0x7f7efc06fe00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8006010 tx=0x7f7ef8005dc0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 msgr2=0x7f7f040a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f040a49c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0665f0 tx=0x7f7f0c069d60 comp rx=0 tx=0).stop 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 shutdown_connections 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7efc06d940 0x7f7efc06fe00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f040941f0 0x7f7f040a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 --2- 172.21.15.39:0/1918932859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f040953f0 0x7f7f040a49c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 >> 172.21.15.39:0/1918932859 conn(0x7f7f0408f770 msgr2=0x7f7f04098620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 shutdown_connections 2024-02-19T19:11:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.290+0000 7f7f119bf700 1 -- 172.21.15.39:0/1918932859 wait complete. 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: pgmap v203: 65 pgs: 65 active+clean; 50 MiB data, 336 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: mds.? [v2:172.21.15.86:6826/1308950780,v1:172.21.15.86:6827/1308950780] up:boot 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 3 up:standby 2024-02-19T19:11:43.424 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:43 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 -- 172.21.15.39:0/1621930020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 msgr2=0x7f8a50100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 --2- 172.21.15.39:0/1621930020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a50100030 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009910 tx=0x7f8a40009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 -- 172.21.15.39:0/1621930020 shutdown_connections 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 --2- 172.21.15.39:0/1621930020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 0x7f8a50101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 --2- 172.21.15.39:0/1621930020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a50100030 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 -- 172.21.15.39:0/1621930020 >> 172.21.15.39:0/1621930020 conn(0x7f8a500fb190 msgr2=0x7f8a500fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 -- 172.21.15.39:0/1621930020 shutdown_connections 2024-02-19T19:11:43.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.427+0000 7f8a56740700 1 -- 172.21.15.39:0/1621930020 wait complete. 2024-02-19T19:11:43.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a56740700 1 Processor -- start 2024-02-19T19:11:43.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a56740700 1 -- start start 2024-02-19T19:11:43.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a56740700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a500690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a56740700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 0x7f8a5019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a56740700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5019b800 con 0x7f8a500ffc10 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a56740700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5019b940 con 0x7f8a50100e10 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a4ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a500690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a4ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a500690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59480/0 (socket says 172.21.15.39:59480) 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.428+0000 7f8a4ffff700 1 -- 172.21.15.39:0/366843385 learned_addr learned my addr 172.21.15.39:0/366843385 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4f7fe700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 0x7f8a5019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4ffff700 1 -- 172.21.15.39:0/366843385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 msgr2=0x7f8a5019b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4ffff700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 0x7f8a5019b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4ffff700 1 -- 172.21.15.39:0/366843385 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400095f0 con 0x7f8a500ffc10 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4f7fe700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 0x7f8a5019b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:11:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4ffff700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a500690e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8a40000c00 tx=0x7f8a4000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:43.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a4001d070 con 0x7f8a500ffc10 2024-02-19T19:11:43.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5019bbc0 con 0x7f8a500ffc10 2024-02-19T19:11:43.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8a40005dc0 con 0x7f8a500ffc10 2024-02-19T19:11:43.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.429+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a400226a0 con 0x7f8a500ffc10 2024-02-19T19:11:43.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.430+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5019c0b0 con 0x7f8a500ffc10 2024-02-19T19:11:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.431+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a50108700 con 0x7f8a500ffc10 2024-02-19T19:11:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.431+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f8a40003910 con 0x7f8a500ffc10 2024-02-19T19:11:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.432+0000 7f8a4d7fa700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8a3806d8a0 0x7f8a3806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.432+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8a4008e930 con 0x7f8a500ffc10 2024-02-19T19:11:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.432+0000 7f8a4f7fe700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8a3806d8a0 0x7f8a3806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.432+0000 7f8a4f7fe700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8a3806d8a0 0x7f8a3806fd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8a50069660 tx=0x7f8a44009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:43.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.439+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8a4005bbc0 con 0x7f8a500ffc10 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: pgmap v203: 65 pgs: 65 active+clean; 50 MiB data, 336 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 158 op/s 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: mds.? [v2:172.21.15.86:6826/1308950780,v1:172.21.15.86:6827/1308950780] up:boot 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 3 up:standby 2024-02-19T19:11:43.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:43 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:11:43.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.795+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8a5019cba0 con 0x7f8a500ffc10 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.796+0000 7f8a4d7fa700 1 -- 172.21.15.39:0/366843385 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8a40060e60 con 0x7f8a500ffc10 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": { 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": { 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": { 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:43.819 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-02-19T19:11:43.820 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:11:43.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-02-19T19:11:43.820 INFO:teuthology.orchestra.run.smithi039.stdout: "overall": { 2024-02-19T19:11:43.820 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 14 2024-02-19T19:11:43.820 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-02-19T19:11:43.820 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8a3806d8a0 msgr2=0x7f8a3806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8a3806d8a0 0x7f8a3806fd60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8a50069660 tx=0x7f8a44009260 comp rx=0 tx=0).stop 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 msgr2=0x7f8a500690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a500690e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8a40000c00 tx=0x7f8a4000f740 comp rx=0 tx=0).stop 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 shutdown_connections 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8a3806d8a0 0x7f8a3806fd60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a500ffc10 0x7f8a500690e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 --2- 172.21.15.39:0/366843385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a50100e10 0x7f8a5019b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 >> 172.21.15.39:0/366843385 conn(0x7f8a500fb190 msgr2=0x7f8a50104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 shutdown_connections 2024-02-19T19:11:43.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.799+0000 7f8a56740700 1 -- 172.21.15.39:0/366843385 wait complete. 2024-02-19T19:11:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 -- 172.21.15.39:0/1605660355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40101750 msgr2=0x7f5f40103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 --2- 172.21.15.39:0/1605660355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40101750 0x7f5f40103b40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5f30009910 tx=0x7f5f30009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 -- 172.21.15.39:0/1605660355 shutdown_connections 2024-02-19T19:11:43.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 --2- 172.21.15.39:0/1605660355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f40104080 0x7f5f40106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 --2- 172.21.15.39:0/1605660355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40101750 0x7f5f40103b40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 -- 172.21.15.39:0/1605660355 >> 172.21.15.39:0/1605660355 conn(0x7f5f400fb160 msgr2=0x7f5f400fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:43.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 -- 172.21.15.39:0/1605660355 shutdown_connections 2024-02-19T19:11:43.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.916+0000 7f5f45592700 1 -- 172.21.15.39:0/1605660355 wait complete. 2024-02-19T19:11:43.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f45592700 1 Processor -- start 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f45592700 1 -- start start 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f45592700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f40101750 0x7f5f401967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f45592700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 0x7f5f40196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f45592700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40197340 con 0x7f5f40104080 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f45592700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40197480 con 0x7f5f40101750 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f3e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 0x7f5f40196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f3e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 0x7f5f40196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59504/0 (socket says 172.21.15.39:59504) 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.917+0000 7f5f3e7fc700 1 -- 172.21.15.39:0/3454765317 learned_addr learned my addr 172.21.15.39:0/3454765317 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f3effd700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f40101750 0x7f5f401967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f3e7fc700 1 -- 172.21.15.39:0/3454765317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f40101750 msgr2=0x7f5f401967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f3e7fc700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f40101750 0x7f5f401967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:43.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f3e7fc700 1 -- 172.21.15.39:0/3454765317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f300095f0 con 0x7f5f40104080 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f3e7fc700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 0x7f5f40196d20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400e920 tx=0x7f5f3400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f3400c9d0 con 0x7f5f40104080 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4019bf30 con 0x7f5f40104080 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4019c450 con 0x7f5f40104080 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5f34010430 con 0x7f5f40104080 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.918+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f34018650 con 0x7f5f40104080 2024-02-19T19:11:43.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.920+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40190630 con 0x7f5f40104080 2024-02-19T19:11:43.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.920+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f5f340105a0 con 0x7f5f40104080 2024-02-19T19:11:43.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.921+0000 7f5f27fff700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5f2806d8a0 0x7f5f2806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:43.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.921+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5f34014070 con 0x7f5f40104080 2024-02-19T19:11:43.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.921+0000 7f5f3effd700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5f2806d8a0 0x7f5f2806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:43.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.922+0000 7f5f3effd700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5f2806d8a0 0x7f5f2806fd60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000b6f0 tx=0x7f5f30005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:43.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:43.926+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5f34052d20 con 0x7f5f40104080 2024-02-19T19:11:44.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.225+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5f40068af0 con 0x7f5f40104080 2024-02-19T19:11:44.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.225+0000 7f5f27fff700 1 -- 172.21.15.39:0/3454765317 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 34 v34) v1 ==== 76+0+1866 (secure 0 0 0) 0x7f5f3405aa80 con 0x7f5f40104080 2024-02-19T19:11:44.248 INFO:teuthology.orchestra.run.smithi039.stdout:e34 2024-02-19T19:11:44.248 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:11:44.248 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 33 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-02-19T19:01:58.803530+0000 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-02-19T19:11:38.148927+0000 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 60 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-02-19T19:11:44.249 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 85 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 1 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:in 0 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=34380} 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3] 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data enabled 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-02-19T19:11:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 1 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.omsmsr{0:34380} state up:active seq 20 join_fscid=1 addr [v2:172.21.15.39:6826/504944654,v1:172.21.15.39:6827/504944654] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi039.aygwnv{-1:34412} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.39:6828/3151633387,v1:172.21.15.39:6829/3151633387] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.euvbgw{-1:44343} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6824/3342991439,v1:172.21.15.86:6825/3342991439] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:11:44.251 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.cephfs.smithi086.idtugy{-1:44349} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.86:6826/1308950780,v1:172.21.15.86:6827/1308950780] compat {c=[1],r=[1],i=[fff]}] 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5f2806d8a0 msgr2=0x7f5f2806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5f2806d8a0 0x7f5f2806fd60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000b6f0 tx=0x7f5f30005a90 comp rx=0 tx=0).stop 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 msgr2=0x7f5f40196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 0x7f5f40196d20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400e920 tx=0x7f5f3400ec30 comp rx=0 tx=0).stop 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 shutdown_connections 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f5f2806d8a0 0x7f5f2806fd60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f40101750 0x7f5f401967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.227+0000 7f5f45592700 1 --2- 172.21.15.39:0/3454765317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f40104080 0x7f5f40196d20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.228+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 >> 172.21.15.39:0/3454765317 conn(0x7f5f400fb160 msgr2=0x7f5f400fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.228+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 shutdown_connections 2024-02-19T19:11:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.228+0000 7f5f45592700 1 -- 172.21.15.39:0/3454765317 wait complete. 2024-02-19T19:11:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 -- 172.21.15.39:0/2025302783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04100cb0 msgr2=0x7f6f041010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 --2- 172.21.15.39:0/2025302783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04100cb0 0x7f6f041010d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009910 tx=0x7f6f00009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 -- 172.21.15.39:0/2025302783 shutdown_connections 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 --2- 172.21.15.39:0/2025302783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04101eb0 0x7f6f04102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 --2- 172.21.15.39:0/2025302783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04100cb0 0x7f6f041010d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 -- 172.21.15.39:0/2025302783 >> 172.21.15.39:0/2025302783 conn(0x7f6f040fc230 msgr2=0x7f6f040fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 -- 172.21.15.39:0/2025302783 shutdown_connections 2024-02-19T19:11:44.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.339+0000 7f6f0c266700 1 -- 172.21.15.39:0/2025302783 wait complete. 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0c266700 1 Processor -- start 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0c266700 1 -- start start 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0c266700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04100cb0 0x7f6f041964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0c266700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 0x7f6f04196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0c266700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04197020 con 0x7f6f04101eb0 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0c266700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04197160 con 0x7f6f04100cb0 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f09801700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 0x7f6f04196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f09801700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 0x7f6f04196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59526/0 (socket says 172.21.15.39:59526) 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f09801700 1 -- 172.21.15.39:0/3501642783 learned_addr learned my addr 172.21.15.39:0/3501642783 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.340+0000 7f6f0a002700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04100cb0 0x7f6f041964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6f09801700 1 -- 172.21.15.39:0/3501642783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04100cb0 msgr2=0x7f6f041964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6f09801700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04100cb0 0x7f6f041964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6f09801700 1 -- 172.21.15.39:0/3501642783 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000095f0 con 0x7f6f04101eb0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6f09801700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 0x7f6f04196a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400e920 tx=0x7f6ef400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ef400c9d0 con 0x7f6f04101eb0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ef4010430 con 0x7f6f04101eb0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0419bc10 con 0x7f6f04101eb0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ef4018650 con 0x7f6f04101eb0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.341+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0419c160 con 0x7f6f04101eb0 2024-02-19T19:11:44.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.342+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee8005320 con 0x7f6f04101eb0 2024-02-19T19:11:44.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.343+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f6ef400cb30 con 0x7f6f04101eb0 2024-02-19T19:11:44.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.344+0000 7f6efaffd700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6ef006d8a0 0x7f6ef006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:44.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.344+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f6ef4014070 con 0x7f6f04101eb0 2024-02-19T19:11:44.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.344+0000 7f6f0a002700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6ef006d8a0 0x7f6ef006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:44.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.344+0000 7f6f0a002700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6ef006d8a0 0x7f6ef006fd60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000b6f0 tx=0x7f6f00005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:44.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.348+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6ef4052d20 con 0x7f6f04101eb0 2024-02-19T19:11:44.498 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:44 smithi039 ceph-mon[92224]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:44.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:44 smithi039 ceph-mon[92224]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:44.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:44 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:44.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:44 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/366843385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:44.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:44 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3454765317' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:11:44.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.583+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 --> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ee8000bf0 con 0x7f6ef006d8a0 2024-02-19T19:11:44.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.585+0000 7f6efaffd700 1 -- 172.21.15.39:0/3501642783 <== mgr.34104 v2:172.21.15.39:6800/517499402 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6ee8000bf0 con 0x7f6ef006d8a0 2024-02-19T19:11:44.607 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-02-19T19:11:44.607 INFO:teuthology.orchestra.run.smithi039.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "in_progress": true, 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "services_complete": [ 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "mon", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "osd", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "crash" 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "progress": "15/23 daemons upgraded", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "message": "Currently upgrading mds daemons", 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout: "is_paused": false 2024-02-19T19:11:44.608 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6ef006d8a0 msgr2=0x7f6ef006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6ef006d8a0 0x7f6ef006fd60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000b6f0 tx=0x7f6f00005940 comp rx=0 tx=0).stop 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 msgr2=0x7f6f04196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 0x7f6f04196a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400e920 tx=0x7f6ef400ec30 comp rx=0 tx=0).stop 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 shutdown_connections 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f6ef006d8a0 0x7f6ef006fd60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04100cb0 0x7f6f041964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 --2- 172.21.15.39:0/3501642783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f04101eb0 0x7f6f04196a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 >> 172.21.15.39:0/3501642783 conn(0x7f6f040fc230 msgr2=0x7f6f041050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 shutdown_connections 2024-02-19T19:11:44.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.587+0000 7f6f0c266700 1 -- 172.21.15.39:0/3501642783 wait complete. 2024-02-19T19:11:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.705+0000 7fc9550bf700 1 -- 172.21.15.39:0/3759082418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950100b70 msgr2=0x7fc950100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.705+0000 7fc9550bf700 1 --2- 172.21.15.39:0/3759082418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950100b70 0x7fc950100f90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc940009910 tx=0x7fc940009c20 comp rx=0 tx=0).stop 2024-02-19T19:11:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 -- 172.21.15.39:0/3759082418 shutdown_connections 2024-02-19T19:11:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 --2- 172.21.15.39:0/3759082418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950101d70 0x7fc9501021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 --2- 172.21.15.39:0/3759082418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950100b70 0x7fc950100f90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 -- 172.21.15.39:0/3759082418 >> 172.21.15.39:0/3759082418 conn(0x7fc9500fc0d0 msgr2=0x7fc9500fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 -- 172.21.15.39:0/3759082418 shutdown_connections 2024-02-19T19:11:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 -- 172.21.15.39:0/3759082418 wait complete. 2024-02-19T19:11:44.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.706+0000 7fc9550bf700 1 Processor -- start 2024-02-19T19:11:44.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc9550bf700 1 -- start start 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc9550bf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950100b70 0x7fc950196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc9550bf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 0x7fc950196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc9550bf700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc950196e80 con 0x7fc950101d70 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc9550bf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc950196fc0 con 0x7fc950100b70 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 0x7fc950196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 0x7fc950196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59542/0 (socket says 172.21.15.39:59542) 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94e59c700 1 -- 172.21.15.39:0/738992503 learned_addr learned my addr 172.21.15.39:0/738992503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94ed9d700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950100b70 0x7fc950196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94e59c700 1 -- 172.21.15.39:0/738992503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950100b70 msgr2=0x7fc950196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94e59c700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950100b70 0x7fc950196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94e59c700 1 -- 172.21.15.39:0/738992503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400095f0 con 0x7fc950101d70 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.707+0000 7fc94ed9d700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950100b70 0x7fc950196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.708+0000 7fc94e59c700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 0x7fc950196860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc94400e920 tx=0x7fc94400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.708+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc94400c9d0 con 0x7fc950101d70 2024-02-19T19:11:44.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.708+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc944010430 con 0x7fc950101d70 2024-02-19T19:11:44.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.708+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95019ba70 con 0x7fc950101d70 2024-02-19T19:11:44.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.708+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc944018650 con 0x7fc950101d70 2024-02-19T19:11:44.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.708+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95019bfc0 con 0x7fc950101d70 2024-02-19T19:11:44.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.709+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9501096d0 con 0x7fc950101d70 2024-02-19T19:11:44.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.710+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fc9440105a0 con 0x7fc950101d70 2024-02-19T19:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.710+0000 7fc937fff700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc93806d8a0 0x7fc93806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.710+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc944014070 con 0x7fc950101d70 2024-02-19T19:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.711+0000 7fc94ed9d700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc93806d8a0 0x7fc93806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.711+0000 7fc94ed9d700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc93806d8a0 0x7fc93806fd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc94000b6f0 tx=0x7fc940005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:44.716+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc944052d20 con 0x7fc950101d70 2024-02-19T19:11:44.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:44 smithi086 ceph-mon[82675]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:44.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:44 smithi086 ceph-mon[82675]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:44.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:44 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:44.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:44 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/366843385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:44.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:44 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3454765317' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:11:45.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.031+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc95019ca00 con 0x7fc950101d70 2024-02-19T19:11:45.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.032+0000 7fc937fff700 1 -- 172.21.15.39:0/738992503 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fc94405ab10 con 0x7fc950101d70 2024-02-19T19:11:45.055 INFO:teuthology.orchestra.run.smithi039.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:11:45.055 INFO:teuthology.orchestra.run.smithi039.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:11:45.055 INFO:teuthology.orchestra.run.smithi039.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc93806d8a0 msgr2=0x7fc93806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc93806d8a0 0x7fc93806fd60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc94000b6f0 tx=0x7fc940005940 comp rx=0 tx=0).stop 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 msgr2=0x7fc950196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 0x7fc950196860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc94400e920 tx=0x7fc94400ec30 comp rx=0 tx=0).stop 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 shutdown_connections 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc93806d8a0 0x7fc93806fd60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc950100b70 0x7fc950196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 --2- 172.21.15.39:0/738992503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc950101d70 0x7fc950196860 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:45.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 >> 172.21.15.39:0/738992503 conn(0x7fc9500fc0d0 msgr2=0x7fc950104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:45.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 shutdown_connections 2024-02-19T19:11:45.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:45.034+0000 7fc9550bf700 1 -- 172.21.15.39:0/738992503 wait complete. 2024-02-19T19:11:45.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:45 smithi039 ceph-mon[92224]: pgmap v204: 65 pgs: 65 active+clean; 61 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 411 op/s 2024-02-19T19:11:45.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:45 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/738992503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:11:45.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:45 smithi086 ceph-mon[82675]: pgmap v204: 65 pgs: 65 active+clean; 61 MiB data, 393 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 411 op/s 2024-02-19T19:11:45.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:45 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/738992503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:11:46.748 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:46 smithi039 ceph-mon[92224]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:46.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:46 smithi086 ceph-mon[82675]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:11:47.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:47 smithi039 ceph-mon[92224]: pgmap v205: 65 pgs: 65 active+clean; 81 MiB data, 482 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 582 op/s 2024-02-19T19:11:47.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:47 smithi086 ceph-mon[82675]: pgmap v205: 65 pgs: 65 active+clean; 81 MiB data, 482 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 582 op/s 2024-02-19T19:11:49.418 INFO:tasks.workunit.client.0.smithi039.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-19T19:11:49.432 INFO:tasks.workunit.client.0.smithi039.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-19T19:11:49.432 INFO:tasks.workunit.client.0.smithi039.stderr:+ make 2024-02-19T19:11:49.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:49 smithi086 ceph-mon[82675]: pgmap v206: 65 pgs: 65 active+clean; 108 MiB data, 576 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 848 op/s 2024-02-19T19:11:49.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:49 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:49.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:49 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:49.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:49 smithi039 ceph-mon[92224]: pgmap v206: 65 pgs: 65 active+clean; 108 MiB data, 576 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 6.3 MiB/s wr, 848 op/s 2024-02-19T19:11:49.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:49 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:49.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:49 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:50.786 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:50 smithi086 ceph-mon[82675]: pgmap v207: 65 pgs: 65 active+clean; 113 MiB data, 615 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 6.3 MiB/s wr, 844 op/s 2024-02-19T19:11:50.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:50.787 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:50 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:50.977 INFO:tasks.workunit.client.0.smithi039.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-19T19:11:50.993 INFO:tasks.workunit.client.0.smithi039.stderr:In file included from fsstress.c:33: 2024-02-19T19:11:50.994 INFO:tasks.workunit.client.0.smithi039.stderr:global.h:1:1: warning: null character(s) ignored 2024-02-19T19:11:50.994 INFO:tasks.workunit.client.0.smithi039.stderr: 1 | 2024-02-19T19:11:50.994 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-02-19T19:11:50.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:50 smithi039 ceph-mon[92224]: pgmap v207: 65 pgs: 65 active+clean; 113 MiB data, 615 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 6.3 MiB/s wr, 844 op/s 2024-02-19T19:11:50.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:50.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:50 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:195:22: error: ‘NULL’ undeclared here (not in a function) 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: 195 | { 0, 0, 'd', NULL }, 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:34:1: note: ‘NULL’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | #include "global.h" 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: +++ |+#include 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: 34 | 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:213:1: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: 213 | off64_t maxfsize; 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.001 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:223:1: error: unknown type name ‘ino_t’; did you mean ‘int’? 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: 223 | ino_t top_ino; 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: | int 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:240:34: error: unknown type name ‘mode_t’ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: 240 | int creat_path(pathname_t *, mode_t); 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:254:35: error: unknown type name ‘uid_t’ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: 254 | int lchown_path(pathname_t *, uid_t, gid_t); 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:254:42: error: unknown type name ‘gid_t’ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: 254 | int lchown_path(pathname_t *, uid_t, gid_t); 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:256:43: warning: ‘struct stat64’ declared inside parameter list will not be visible outside of this definition or declaration 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: 256 | int lstat64_path(pathname_t *, struct stat64 *); 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.002 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:258:34: error: unknown type name ‘mode_t’ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: 258 | int mkdir_path(pathname_t *, mode_t); 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:259:34: error: unknown type name ‘mode_t’ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: 259 | int mknod_path(pathname_t *, mode_t, dev_t); 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:259:42: error: unknown type name ‘dev_t’ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: 259 | int mknod_path(pathname_t *, mode_t, dev_t); 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:262:1: error: unknown type name ‘DIR’ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: 262 | DIR *opendir_path(pathname_t *); 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:264:45: error: unknown type name ‘size_t’ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: 264 | int readlink_path(pathname_t *, char *, size_t); 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:264:45: note: ‘size_t’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.003 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:269:42: warning: ‘struct stat64’ declared inside parameter list will not be visible outside of this definition or declaration 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr: 269 | int stat64_path(pathname_t *, struct stat64 *); 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:271:39: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr: 271 | int truncate64_path(pathname_t *, off64_t); 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘main’: 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:293:25: error: storage size of ‘t’ isn’t known 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr: 293 | struct timeval t; 2024-02-19T19:11:51.004 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:303:9: warning: implicit declaration of function ‘umask’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr: 303 | umask(0); 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:307:21: warning: implicit declaration of function ‘getopt’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr: 307 | while ((c = getopt(argc, argv, "cd:e:f:i:l:n:p:rs:vwzHSX")) != -1) { 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:314:35: error: ‘optarg’ undeclared (first use in this function) 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr: 314 | dirname = optarg; 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:314:35: note: each undeclared identifier is reported only once for each function it appears in 2024-02-19T19:11:51.008 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:317:25: warning: implicit declaration of function ‘sscanf’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: 317 | sscanf(optarg, "%d", &errtag); 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:34:1: note: include ‘’ or provide a declaration of ‘sscanf’ 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | #include "global.h" 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: +++ |+#include 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: 34 | 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:317:25: warning: incompatible implicit declaration of built-in function ‘sscanf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: 317 | sscanf(optarg, "%d", &errtag); 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:317:25: note: include ‘’ or provide a declaration of ‘sscanf’ 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:324:33: warning: implicit declaration of function ‘fprintf’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: 324 | fprintf(stderr, 2024-02-19T19:11:51.009 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:324:33: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:324:33: warning: incompatible implicit declaration of built-in function ‘fprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:324:33: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:324:41: error: ‘stderr’ undeclared (first use in this function) 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: 324 | fprintf(stderr, 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:324:41: note: ‘stderr’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:327:33: warning: implicit declaration of function ‘exit’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: 327 | exit(1); 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:34:1: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | #include "global.h" 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: +++ |+#include 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr: 34 | 2024-02-19T19:11:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:327:33: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: 327 | exit(1); 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:327:33: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:334:33: warning: implicit declaration of function ‘realloc’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: 334 | ilist = realloc(ilist, ++ilistlen * sizeof(*ilist)); 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:334:33: note: include ‘’ or provide a declaration of ‘realloc’ 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:334:33: warning: incompatible implicit declaration of built-in function ‘realloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:334:33: note: include ‘’ or provide a declaration of ‘realloc’ 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:335:47: warning: implicit declaration of function ‘strtol’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: 335 | ilist[ilistlen - 1] = strtol(optarg, &p, 16); 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:338:33: warning: implicit declaration of function ‘atoi’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.011 INFO:tasks.workunit.client.0.smithi039.stderr: 338 | loops = atoi(optarg); 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:350:32: warning: implicit declaration of function ‘strtoul’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: 350 | seed = strtoul(optarg, NULL, 0); 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:363:25: warning: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: 363 | printf("\n"); 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:363:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:363:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:363:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:367:25: warning: incompatible implicit declaration of built-in function ‘fprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: 367 | fprintf(stderr, "%s - invalid parameters\n", 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.012 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:367:25: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:372:25: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr: 372 | exit(1); 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:372:25: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:380:17: warning: incompatible implicit declaration of built-in function ‘fprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr: 380 | fprintf(stderr, "error injection only works on XFS\n"); 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:380:17: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:381:17: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr: 381 | exit(1); 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:381:17: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.013 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:399:25: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.014 INFO:tasks.workunit.client.0.smithi039.stderr: 399 | exit(1); 2024-02-19T19:11:51.014 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.014 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:399:25: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:402:23: warning: implicit declaration of function ‘mkdir’; did you mean ‘mkdir_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr: 402 | (void)mkdir(dirname, 0777); 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr: | mkdir_f 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:403:21: warning: implicit declaration of function ‘chdir’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr: 403 | if (chdir(dirname) < 0) { 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:404:25: warning: implicit declaration of function ‘perror’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.037 INFO:tasks.workunit.client.0.smithi039.stderr: 404 | perror(dirname); 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:405:25: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr: 405 | exit(1); 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:405:25: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:407:17: warning: implicit declaration of function ‘sprintf’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr: 407 | sprintf(buf, "fss%x", getpid()); 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:407:17: note: include ‘’ or provide a declaration of ‘sprintf’ 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:407:17: warning: incompatible implicit declaration of built-in function ‘sprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:407:17: note: include ‘’ or provide a declaration of ‘sprintf’ 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:407:39: warning: implicit declaration of function ‘getpid’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.038 INFO:tasks.workunit.client.0.smithi039.stderr: 407 | sprintf(buf, "fss%x", getpid()); 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:408:22: warning: implicit declaration of function ‘creat’; did you mean ‘creat_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: 408 | fd = creat(buf, 0666); 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: | creat_f 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:409:21: warning: implicit declaration of function ‘lseek64’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: 409 | if (lseek64(fd, (off64_t)(MAXFSIZE32 + 1ULL), SEEK_SET) < 0) 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:409:34: error: ‘off64_t’ undeclared (first use in this function) 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: 409 | if (lseek64(fd, (off64_t)(MAXFSIZE32 + 1ULL), SEEK_SET) < 0) 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:409:63: error: ‘SEEK_SET’ undeclared (first use in this function) 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: 409 | if (lseek64(fd, (off64_t)(MAXFSIZE32 + 1ULL), SEEK_SET) < 0) 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.039 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:414:17: warning: implicit declaration of function ‘setlinebuf’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: 414 | setlinebuf(stdout); 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:414:28: error: ‘stdout’ undeclared (first use in this function) 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: 414 | setlinebuf(stdout); 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:414:28: note: ‘stdout’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:416:25: warning: implicit declaration of function ‘gettimeofday’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: 416 | gettimeofday(&t, (void *)NULL); 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:418:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: 418 | printf("seed = %ld\n", seed); 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:418:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.040 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:461:25: warning: implicit declaration of function ‘close’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: 461 | close(fd); 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:462:17: warning: implicit declaration of function ‘unlink’; did you mean ‘unlink_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: 462 | unlink(buf); 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: | unlink_f 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:468:37: warning: implicit declaration of function ‘fork’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: 468 | if (fork() == 0) { 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:474:32: warning: implicit declaration of function ‘wait’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: 474 | while (wait(&stat) > 0) 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:493:25: warning: implicit declaration of function ‘system’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.041 INFO:tasks.workunit.client.0.smithi039.stderr: 493 | system(cmd); 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:497:33: warning: implicit declaration of function ‘free’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr: 497 | free(flist[i].fents); 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:497:33: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:497:33: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:497:33: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘add_to_flist’: 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:515:30: warning: incompatible implicit declaration of built-in function ‘realloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr: 515 | ftp->fents = realloc(ftp->fents, ftp->nslots * sizeof(fent_t)); 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:515:30: note: include ‘’ or provide a declaration of ‘realloc’ 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘append_pathname’: 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:527:15: warning: implicit declaration of function ‘strlen’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.042 INFO:tasks.workunit.client.0.smithi039.stderr: 527 | len = strlen(str); 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:34:1: note: include ‘’ or provide a declaration of ‘strlen’ 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | #include "global.h" 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: +++ |+#include 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: 34 | 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:527:15: warning: incompatible implicit declaration of built-in function ‘strlen’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: 527 | len = strlen(str); 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:527:15: note: include ‘’ or provide a declaration of ‘strlen’ 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:536:22: warning: incompatible implicit declaration of built-in function ‘realloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: 536 | name->path = realloc(name->path, name->len + 1 + len); 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:536:22: note: include ‘’ or provide a declaration of ‘realloc’ 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:537:9: warning: implicit declaration of function ‘strcpy’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.043 INFO:tasks.workunit.client.0.smithi039.stderr: 537 | strcpy(&name->path[name->len], str); 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:537:9: note: include ‘’ or provide a declaration of ‘strcpy’ 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:537:9: warning: incompatible implicit declaration of built-in function ‘strcpy’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:537:9: note: include ‘’ or provide a declaration of ‘strcpy’ 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:620:30: error: unknown type name ‘mode_t’ 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr: 620 | creat_path(pathname_t *name, mode_t mode) 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘doproc’: 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:713:25: error: storage size of ‘statbuf’ isn’t known 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr: 713 | struct stat64 statbuf; 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:719:9: warning: incompatible implicit declaration of built-in function ‘sprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.044 INFO:tasks.workunit.client.0.smithi039.stderr: 719 | sprintf(buf, "p%x", procid); 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:719:9: note: include ‘’ or provide a declaration of ‘sprintf’ 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:721:31: warning: implicit declaration of function ‘stat64’; did you mean ‘stat_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: 721 | if (chdir(buf) < 0 || stat64(".", &statbuf) < 0) { 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: | stat_f 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:723:17: warning: implicit declaration of function ‘_exit’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: 723 | _exit(1); 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:723:17: warning: incompatible implicit declaration of built-in function ‘_exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:726:19: warning: implicit declaration of function ‘getcwd’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: 726 | homedir = getcwd(NULL, -1); 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.045 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:728:9: warning: implicit declaration of function ‘srandom’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: 728 | srandom(seed); 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:730:28: warning: implicit declaration of function ‘random’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: 730 | namerand = random(); 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:733:52: warning: implicit declaration of function ‘abort’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: 733 | if ((unsigned long)p->func < 4096) abort(); 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:733:52: note: include ‘’ or provide a declaration of ‘abort’ 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:733:52: warning: incompatible implicit declaration of built-in function ‘abort’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:733:52: note: include ‘’ or provide a declaration of ‘abort’ 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:744:37: error: ‘EIO’ undeclared (first use in this function) 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: 744 | if (rval == EIO) { 2024-02-19T19:11:51.046 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:745:33: warning: incompatible implicit declaration of built-in function ‘fprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: 745 | fprintf(stderr, "Detected EIO\n"); 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:745:33: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:745:41: error: ‘stderr’ undeclared (first use in this function) 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: 745 | fprintf(stderr, "Detected EIO\n"); 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:745:41: note: ‘stderr’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘fent_to_name’: 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:755:21: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: 755 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:766:13: warning: incompatible implicit declaration of built-in function ‘sprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: 766 | i = sprintf(buf, "%c%x", flp->tag, fep->id); 2024-02-19T19:11:51.047 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:766:13: note: include ‘’ or provide a declaration of ‘sprintf’ 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘free_pathname’: 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:791:17: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr: 791 | free(name->path); 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:791:17: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘generate_fname’: 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:800:21: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr: 800 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:807:15: warning: incompatible implicit declaration of built-in function ‘sprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr: 807 | len = sprintf(buf, "%c%x", flp->tag, id = nameseq++); 2024-02-19T19:11:51.048 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:807:15: note: include ‘’ or provide a declaration of ‘sprintf’ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:887:31: error: unknown type name ‘uid_t’ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: 887 | lchown_path(pathname_t *name, uid_t owner, gid_t group) 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:887:44: error: unknown type name ‘gid_t’ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: 887 | lchown_path(pathname_t *name, uid_t owner, gid_t group) 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘link_path’: 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:908:30: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: 908 | char buf1[MAXNAMELEN]; 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:915:16: warning: implicit declaration of function ‘link’; did you mean ‘link_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: 915 | rval = link(name1->path, name2->path); 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.049 INFO:tasks.workunit.client.0.smithi039.stderr: | link_f 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:916:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: 916 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:34:1: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | #include "global.h" 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: +++ |+#include 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: 34 | 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:916:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: 916 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:920:13: warning: implicit declaration of function ‘strcmp’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: 920 | if (strcmp(buf1, buf2) == 0) { 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:920:13: note: include ‘’ or provide a declaration of ‘strcmp’ 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:930:26: warning: incompatible implicit declaration of built-in function ‘strlen’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.050 INFO:tasks.workunit.client.0.smithi039.stderr: 930 | else if (strlen(buf1) == 0) 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:930:26: note: include ‘’ or provide a declaration of ‘strlen’ 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:935:33: warning: implicit declaration of function ‘MAX’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: 935 | down1 = MAX(newname1.len, 3 + name2->len) <= 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:961:39: warning: ‘struct stat64’ declared inside parameter list will not be visible outside of this definition or declaration 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: 961 | lstat64_path(pathname_t *name, struct stat64 *sbuf) 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:961:1: error: conflicting types for ‘lstat64_path’; have ‘int(pathname_t *, struct stat64 *)’ {aka ‘int(struct pathname *, struct stat64 *)’} 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: 961 | lstat64_path(pathname_t *name, struct stat64 *sbuf) 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:256:9: note: previous declaration of ‘lstat64_path’ with type ‘int(pathname_t *, struct stat64 *)’ {aka ‘int(struct pathname *, struct stat64 *)’} 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: 256 | int lstat64_path(pathname_t *, struct stat64 *); 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.051 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘lstat64_path’: 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:963:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: 963 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:967:16: warning: implicit declaration of function ‘lstat64’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: 967 | rval = lstat64(name->path, sbuf); 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:968:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: 968 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:968:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:968:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: 968 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘make_freq_table’: 2024-02-19T19:11:51.052 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:988:22: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: 988 | freq_table = malloc(f * sizeof(*freq_table)); 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:988:22: note: include ‘’ or provide a declaration of ‘malloc’ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:988:22: warning: incompatible implicit declaration of built-in function ‘malloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:988:22: note: include ‘’ or provide a declaration of ‘malloc’ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:997:30: error: unknown type name ‘mode_t’ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: 997 | mkdir_path(pathname_t *name, mode_t mode) 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1016:30: error: unknown type name ‘mode_t’ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: 1016 | mknod_path(pathname_t *name, mode_t mode, dev_t dev) 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1016:43: error: unknown type name ‘dev_t’ 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: 1016 | mknod_path(pathname_t *name, mode_t mode, dev_t dev) 2024-02-19T19:11:51.053 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘namerandpad’: 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1039:61: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr: 1039 | { 2, 4, 8, 16, 32, 64, 128, MAXNAMELEN - 1 }; 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1051:17: warning: implicit declaration of function ‘memset’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr: 1051 | memset(&buf[i], 'X', padlen); 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1051:17: note: include ‘’ or provide a declaration of ‘memset’ 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1051:17: warning: incompatible implicit declaration of built-in function ‘memset’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1051:17: note: include ‘’ or provide a declaration of ‘memset’ 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘open_path’: 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1059:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr: 1059 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.054 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1063:16: warning: implicit declaration of function ‘open’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: 1063 | rval = open(name->path, oflag); 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1064:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: 1064 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1064:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1064:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: 1064 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1075:1: error: unknown type name ‘DIR’ 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: 1075 | DIR * 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.055 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘opendir_path’: 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1078:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: 1078 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1080:9: error: unknown type name ‘DIR’ 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: 1080 | DIR *rval; 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1082:16: warning: implicit declaration of function ‘opendir’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: 1082 | rval = opendir(name->path); 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1082:14: warning: assignment to ‘int *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion] 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: 1082 | rval = opendir(name->path); 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1083:21: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.056 INFO:tasks.workunit.client.0.smithi039.stderr: 1083 | if (rval || errno != ENAMETOOLONG) 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1083:21: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1083:30: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: 1083 | if (rval || errno != ENAMETOOLONG) 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘process_freq’: 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1100:13: warning: implicit declaration of function ‘strchr’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: 1100 | s = strchr(arg, '='); 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1100:13: note: include ‘’ or provide a declaration of ‘strchr’ 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1100:13: warning: incompatible implicit declaration of built-in function ‘strchr’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1100:13: note: include ‘’ or provide a declaration of ‘strchr’ 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1102:17: warning: incompatible implicit declaration of built-in function ‘fprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: 1102 | fprintf(stderr, "bad argument '%s'\n", arg); 2024-02-19T19:11:51.057 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1102:17: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1102:25: error: ‘stderr’ undeclared (first use in this function) 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: 1102 | fprintf(stderr, "bad argument '%s'\n", arg); 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1102:25: note: ‘stderr’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1103:17: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: 1103 | exit(1); 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1103:17: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1112:9: warning: incompatible implicit declaration of built-in function ‘fprintf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: 1112 | fprintf(stderr, "can't find op type %s for -f\n", arg); 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1112:9: note: include ‘’ or provide a declaration of ‘fprintf’ 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1113:9: warning: incompatible implicit declaration of built-in function ‘exit’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.058 INFO:tasks.workunit.client.0.smithi039.stderr: 1113 | exit(1); 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1113:9: note: include ‘’ or provide a declaration of ‘exit’ 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1117:45: error: unknown type name ‘size_t’ 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: 1117 | readlink_path(pathname_t *name, char *lbuf, size_t lbufsiz) 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1117:45: note: ‘size_t’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘rename_path’: 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1138:30: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: 1138 | char buf1[MAXNAMELEN]; 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1145:16: warning: implicit declaration of function ‘rename’; did you mean ‘rename_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: 1145 | rval = rename(name1->path, name2->path); 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.059 INFO:tasks.workunit.client.0.smithi039.stderr: | rename_f 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1146:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr: 1146 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1146:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1146:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr: 1146 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1160:26: warning: incompatible implicit declaration of built-in function ‘strlen’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr: 1160 | else if (strlen(buf1) == 0) 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1160:26: note: include ‘’ or provide a declaration of ‘strlen’ 2024-02-19T19:11:51.060 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘rmdir_path’: 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1193:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: 1193 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1197:16: warning: implicit declaration of function ‘rmdir’; did you mean ‘rmdir_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: 1197 | rval = rmdir(name->path); 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: | rmdir_f 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1198:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: 1198 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1198:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1198:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: 1198 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘separate_pathname’: 2024-02-19T19:11:51.061 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1215:17: warning: incompatible implicit declaration of built-in function ‘strchr’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: 1215 | slash = strchr(name->path, '/'); 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1215:17: note: include ‘’ or provide a declaration of ‘strchr’ 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1221:9: warning: incompatible implicit declaration of built-in function ‘strcpy’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: 1221 | strcpy(buf, name->path); 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1221:9: note: include ‘’ or provide a declaration of ‘strcpy’ 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘show_ops’: 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1238:51: warning: incompatible implicit declaration of built-in function ‘strlen’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: 1238 | if (lead_str != NULL && x+strlen(p->name)>=WIDTH-5) 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1238:51: note: include ‘’ or provide a declaration of ‘strlen’ 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1239:35: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.062 INFO:tasks.workunit.client.0.smithi039.stderr: 1239 | x=printf("%s%s", (p==ops)?"":"\n", lead_str); 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1239:35: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1240:28: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: 1240 | x+=printf("%s ", p->name); 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1240:28: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1242:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: 1242 | printf("\n"); 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1242:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1254:41: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: 1254 | printf("%s", lead_str); 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.063 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1254:41: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1255:33: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: 1255 | printf("%20s %d/%d %s\n", 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1255:33: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1264:38: warning: ‘struct stat64’ declared inside parameter list will not be visible outside of this definition or declaration 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: 1264 | stat64_path(pathname_t *name, struct stat64 *sbuf) 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1264:1: error: conflicting types for ‘stat64_path’; have ‘int(pathname_t *, struct stat64 *)’ {aka ‘int(struct pathname *, struct stat64 *)’} 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: 1264 | stat64_path(pathname_t *name, struct stat64 *sbuf) 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~ 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:269:9: note: previous declaration of ‘stat64_path’ with type ‘int(pathname_t *, struct stat64 *)’ {aka ‘int(struct pathname *, struct stat64 *)’} 2024-02-19T19:11:51.064 INFO:tasks.workunit.client.0.smithi039.stderr: 269 | int stat64_path(pathname_t *, struct stat64 *); 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~ 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘stat64_path’: 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1266:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: 1266 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1271:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: 1271 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1271:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1271:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: 1271 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘symlink_path’: 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1285:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.065 INFO:tasks.workunit.client.0.smithi039.stderr: 1285 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1290:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: 1290 | printf("yikes! %s %s\n", name1, name->path); 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1290:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1294:16: warning: implicit declaration of function ‘symlink’; did you mean ‘symlink_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: 1294 | rval = symlink(name1, name->path); 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: | symlink_f 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1295:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: 1295 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1295:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.066 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1295:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: 1295 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: At top level: 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1307:35: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: 1307 | truncate64_path(pathname_t *name, off64_t length) 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘unlink_path’: 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1328:29: error: ‘MAXNAMELEN’ undeclared (first use in this function) 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: 1328 | char buf[MAXNAMELEN]; 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1333:26: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: 1333 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1333:26: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.067 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1333:35: error: ‘ENAMETOOLONG’ undeclared (first use in this function) 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: 1333 | if (rval >= 0 || errno != ENAMETOOLONG) 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘usage’: 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1347:9: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: 1347 | printf("Usage: %s -H or\n", myprog); 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1347:9: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘chown_f’: 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1642:9: error: unknown type name ‘uid_t’ 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: 1642 | uid_t u; 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1648:14: error: ‘uid_t’ undeclared (first use in this function) 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: 1648 | u = (uid_t)random(); 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.068 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1648:20: error: expected ‘;’ before ‘random’ 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: 1648 | u = (uid_t)random(); 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: | ; 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1651:13: warning: implicit declaration of function ‘lchown_path’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: 1651 | e = lchown_path(&f, u, -1) < 0 ? errno : 0; 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~ 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1651:42: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: 1651 | e = lchown_path(&f, u, -1) < 0 ? errno : 0; 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1651:42: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1654:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: 1654 | printf("%d/%d: chown %s %d %d\n", procid, opno, f.path, u, e); 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.069 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1654:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘creat_f’: 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1689:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: 1689 | printf("%d/%d: creat - no filename from %s\n", 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1689:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1695:14: warning: implicit declaration of function ‘creat_path’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: 1695 | fd = creat_path(&f, 0666); 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1696:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: 1696 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1696:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1717:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.070 INFO:tasks.workunit.client.0.smithi039.stderr: 1717 | printf("%d/%d: creat %s x:%d %d %d\n", procid, opno, f.path, 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1717:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘setdirect’: 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1733:17: warning: implicit declaration of function ‘fcntl’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: 1733 | flags = fcntl(fd, F_GETFL, 0); 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1733:27: error: ‘F_GETFL’ undeclared (first use in this function) 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: 1733 | flags = fcntl(fd, F_GETFL, 0); 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1737:23: error: ‘F_SETFL’ undeclared (first use in this function) 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: 1737 | if (fcntl(fd, F_SETFL, flags|O_DIRECT) < 0) { 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1737:38: error: ‘O_DIRECT’ undeclared (first use in this function) 2024-02-19T19:11:51.071 INFO:tasks.workunit.client.0.smithi039.stderr: 1737 | if (fcntl(fd, F_SETFL, flags|O_DIRECT) < 0) { 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1742:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: 1742 | printf("cannot set O_DIRECT: %s\n",strerror(errno)); 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1742:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1742:52: warning: implicit declaration of function ‘strerror’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: 1742 | printf("cannot set O_DIRECT: %s\n",strerror(errno)); 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1742:61: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: 1742 | printf("cannot set O_DIRECT: %s\n",strerror(errno)); 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1742:61: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘dread_f’: 2024-02-19T19:11:51.072 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1752:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: 1752 | __int64_t align; 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1754:25: error: storage size of ‘diob’ isn’t known 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: 1754 | struct dioattr diob; 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1758:9: error: unknown type name ‘size_t’ 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: 1758 | size_t len; 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1758:9: note: ‘size_t’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1759:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: 1759 | __int64_t lr; 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.073 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1760:9: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: 1760 | off64_t off; 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1761:25: error: storage size of ‘stb’ isn’t known 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: 1761 | struct stat64 stb; 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1767:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: 1767 | printf("%d/%d: dread - no filename\n", procid, opno); 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1767:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1771:28: error: ‘O_RDONLY’ undeclared (first use in this function) 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: 1771 | fd = open_path(&f, O_RDONLY); 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1773:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.074 INFO:tasks.workunit.client.0.smithi039.stderr: 1773 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1773:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1777:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: 1777 | printf("%d/%d: dread - open %s failed %d\n", 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1777:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1789:13: warning: implicit declaration of function ‘fstat64’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: 1789 | if (fstat64(fd, &stb) < 0) { 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1791:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: 1791 | printf("%d/%d: dread - fstat64 %s failed %d\n", 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1791:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1799:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.075 INFO:tasks.workunit.client.0.smithi039.stderr: 1799 | printf("%d/%d: dread - %s zero size\n", procid, opno, 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1799:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1822:18: error: ‘__int64_t’ undeclared (first use in this function); did you mean ‘__int128_t’? 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: 1822 | align = (__int64_t)diob.d_miniosz; 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1822:28: error: expected ‘;’ before ‘diob’ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: 1822 | align = (__int64_t)diob.d_miniosz; 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | ; 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1823:26: error: expected ‘)’ before ‘random’ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: 1823 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | ~ ^~~~~~ 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr: | ) 2024-02-19T19:11:51.076 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1824:16: error: ‘off64_t’ undeclared (first use in this function) 2024-02-19T19:11:51.077 INFO:tasks.workunit.client.0.smithi039.stderr: 1824 | off = (off64_t)(lr % stb.st_size); 2024-02-19T19:11:51.077 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-19T19:11:51.097 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1826:26: error: ‘SEEK_SET’ undeclared (first use in this function) 2024-02-19T19:11:51.097 INFO:tasks.workunit.client.0.smithi039.stderr: 1826 | lseek64(fd, off, SEEK_SET); 2024-02-19T19:11:51.097 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.097 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1827:28: warning: implicit declaration of function ‘getpagesize’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.097 INFO:tasks.workunit.client.0.smithi039.stderr: 1827 | len = (random() % (getpagesize() * 32)) + 1; 2024-02-19T19:11:51.097 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~ 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1833:15: warning: implicit declaration of function ‘memalign’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: 1833 | buf = memalign(diob.d_mem, len); 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1834:13: warning: implicit declaration of function ‘read’; did you mean ‘read_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: 1834 | e = read(fd, buf, len) < 0 ? errno : 0; 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: | read_f 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1835:9: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: 1835 | free(buf); 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1835:9: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1837:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: 1837 | printf("%d/%d: dread %s [%lld,%ld] %d\n", 2024-02-19T19:11:51.098 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1837:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘dwrite_f’: 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1846:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: 1846 | __int64_t align; 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1848:25: error: storage size of ‘diob’ isn’t known 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: 1848 | struct dioattr diob; 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1852:9: error: unknown type name ‘size_t’ 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: 1852 | size_t len; 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.099 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1852:9: note: ‘size_t’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1853:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: 1853 | __int64_t lr; 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1854:9: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: 1854 | off64_t off; 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1855:25: error: storage size of ‘stb’ isn’t known 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: 1855 | struct stat64 stb; 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1861:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.100 INFO:tasks.workunit.client.0.smithi039.stderr: 1861 | printf("%d/%d: dwrite - no filename\n", procid, opno); 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1861:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1865:28: error: ‘O_WRONLY’ undeclared (first use in this function) 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: 1865 | fd = open_path(&f, O_WRONLY); 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1866:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: 1866 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1866:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1870:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: 1870 | printf("%d/%d: dwrite - open %s failed %d\n", 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1870:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1883:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.101 INFO:tasks.workunit.client.0.smithi039.stderr: 1883 | printf("%d/%d: dwrite - fstat64 %s failed %d\n", 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1883:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1905:18: error: ‘__int64_t’ undeclared (first use in this function); did you mean ‘__int128_t’? 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: 1905 | align = (__int64_t)diob.d_miniosz; 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1905:28: error: expected ‘;’ before ‘diob’ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: 1905 | align = (__int64_t)diob.d_miniosz; 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | ; 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1906:26: error: expected ‘)’ before ‘random’ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: 1906 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | ~ ^~~~~~ 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: | ) 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1907:16: error: ‘off64_t’ undeclared (first use in this function) 2024-02-19T19:11:51.102 INFO:tasks.workunit.client.0.smithi039.stderr: 1907 | off = (off64_t)(lr % MIN(stb.st_size + (1024 * 1024), MAXFSIZE)); 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1907:30: warning: implicit declaration of function ‘MIN’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: 1907 | off = (off64_t)(lr % MIN(stb.st_size + (1024 * 1024), MAXFSIZE)); 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1909:26: error: ‘SEEK_SET’ undeclared (first use in this function) 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: 1909 | lseek64(fd, off, SEEK_SET); 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1919:9: warning: incompatible implicit declaration of built-in function ‘memset’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: 1919 | memset(buf, nameseq & 0xff, len); 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1919:9: note: include ‘’ or provide a declaration of ‘memset’ 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1920:13: warning: implicit declaration of function ‘write’; did you mean ‘write_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: 1920 | e = write(fd, buf, len) < 0 ? errno : 0; 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.103 INFO:tasks.workunit.client.0.smithi039.stderr: | write_f 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1921:9: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr: 1921 | free(buf); 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1921:9: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1923:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr: 1923 | printf("%d/%d: dwrite %s [%lld,%ld] %d\n", 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1923:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘fdatasync_f’: 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1940:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr: 1940 | printf("%d/%d: fdatasync - no filename\n", 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1940:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.104 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1945:28: error: ‘O_WRONLY’ undeclared (first use in this function) 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: 1945 | fd = open_path(&f, O_WRONLY); 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1946:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: 1946 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1946:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1950:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: 1950 | printf("%d/%d: fdatasync - open %s failed %d\n", 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1950:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1955:13: warning: implicit declaration of function ‘fdatasync’; did you mean ‘fdatasync_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: 1955 | e = fdatasync(fd) < 0 ? errno : 0; 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.105 INFO:tasks.workunit.client.0.smithi039.stderr: | fdatasync_f 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1957:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: 1957 | printf("%d/%d: fdatasync %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:1957:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘fsync_f’: 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2027:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: 2027 | printf("%d/%d: fsync - no filename\n", procid, opno); 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2027:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2031:28: error: ‘O_WRONLY’ undeclared (first use in this function) 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: 2031 | fd = open_path(&f, O_WRONLY); 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2032:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.106 INFO:tasks.workunit.client.0.smithi039.stderr: 2032 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2032:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2036:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: 2036 | printf("%d/%d: fsync - open %s failed %d\n", 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2036:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2041:13: warning: implicit declaration of function ‘fsync’; did you mean ‘fsync_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: 2041 | e = fsync(fd) < 0 ? errno : 0; 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: | fsync_f 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2043:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: 2043 | printf("%d/%d: fsync %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.107 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2043:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘getdents_f’: 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2051:9: error: unknown type name ‘DIR’ 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: 2051 | DIR *dir; 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2062:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: 2062 | printf("%d/%d: getdents - can't open %s\n", 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2062:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2067:16: warning: implicit declaration of function ‘readdir64’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: 2067 | while (readdir64(dir) != NULL) 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2070:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.108 INFO:tasks.workunit.client.0.smithi039.stderr: 2070 | printf("%d/%d: getdents %s 0\n", procid, opno, f.path); 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2070:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2072:9: warning: implicit declaration of function ‘closedir’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: 2072 | closedir(dir); 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘link_f’: 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2091:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: 2091 | printf("%d/%d: link - no file\n", procid, opno); 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2091:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2106:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: 2106 | printf("%d/%d: link - no filename from %s\n", 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.109 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2106:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2113:37: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr: 2113 | e = link_path(&f, &l) < 0 ? errno : 0; 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2113:37: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2118:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr: 2118 | printf("%d/%d: link %s %s %d\n", procid, opno, f.path, l.path, 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2118:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘mkdir_f’: 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2145:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr: 2145 | printf("%d/%d: mkdir - no filename from %s\n", 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2145:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.110 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2151:13: warning: implicit declaration of function ‘mkdir_path’; did you mean ‘rmdir_path’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: 2151 | e = mkdir_path(&f, 0777) < 0 ? errno : 0; 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: | rmdir_path 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2151:40: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: 2151 | e = mkdir_path(&f, 0777) < 0 ? errno : 0; 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2151:40: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2156:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: 2156 | printf("%d/%d: mkdir %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2156:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘mknod_f’: 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2181:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.111 INFO:tasks.workunit.client.0.smithi039.stderr: 2181 | printf("%d/%d: mknod - no filename from %s\n", 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2181:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2187:13: warning: implicit declaration of function ‘mknod_path’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: 2187 | e = mknod_path(&f, S_IFCHR|0444, 0) < 0 ? errno : 0; 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2187:28: error: ‘S_IFCHR’ undeclared (first use in this function) 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: 2187 | e = mknod_path(&f, S_IFCHR|0444, 0) < 0 ? errno : 0; 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2187:51: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: 2187 | e = mknod_path(&f, S_IFCHR|0444, 0) < 0 ? errno : 0; 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2187:51: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2192:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.112 INFO:tasks.workunit.client.0.smithi039.stderr: 2192 | printf("%d/%d: mknod %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2192:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘read_f’: 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2203:9: error: unknown type name ‘size_t’ 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: 2203 | size_t len; 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2203:9: note: ‘size_t’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2204:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: 2204 | __int64_t lr; 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2205:9: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: 2205 | off64_t off; 2024-02-19T19:11:51.113 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2206:25: error: storage size of ‘stb’ isn’t known 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: 2206 | struct stat64 stb; 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2212:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: 2212 | printf("%d/%d: read - no filename\n", procid, opno); 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2212:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2216:28: error: ‘O_RDONLY’ undeclared (first use in this function) 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: 2216 | fd = open_path(&f, O_RDONLY); 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2217:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: 2217 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.114 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2217:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2221:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: 2221 | printf("%d/%d: read - open %s failed %d\n", 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2221:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2228:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: 2228 | printf("%d/%d: read - fstat64 %s failed %d\n", 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2228:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2236:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: 2236 | printf("%d/%d: read - %s zero size\n", procid, opno, 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2236:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2242:16: error: ‘__int64_t’ undeclared (first use in this function); did you mean ‘__int128_t’? 2024-02-19T19:11:51.115 INFO:tasks.workunit.client.0.smithi039.stderr: 2242 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2242:26: error: expected ‘)’ before ‘random’ 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: 2242 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: | ~ ^~~~~~ 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: | ) 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2243:16: error: ‘off64_t’ undeclared (first use in this function) 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: 2243 | off = (off64_t)(lr % stb.st_size); 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2244:26: error: ‘SEEK_SET’ undeclared (first use in this function) 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: 2244 | lseek64(fd, off, SEEK_SET); 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.116 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2246:15: warning: incompatible implicit declaration of built-in function ‘malloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: 2246 | buf = malloc(len); 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2246:15: note: include ‘’ or provide a declaration of ‘malloc’ 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2248:9: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: 2248 | free(buf); 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2248:9: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2250:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: 2250 | printf("%d/%d: read %s [%lld,%ld] %d\n", 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2250:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘readlink_f’: 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2259:29: error: ‘PATH_MAX’ undeclared (first use in this function) 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: 2259 | char buf[PATH_MAX]; 2024-02-19T19:11:51.117 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2267:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: 2267 | printf("%d/%d: readlink - no filename\n", procid, opno); 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2267:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2271:13: warning: implicit declaration of function ‘readlink_path’; did you mean ‘unlink_path’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: 2271 | e = readlink_path(&f, buf, PATH_MAX) < 0 ? errno : 0; 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~ 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: | unlink_path 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2271:52: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: 2271 | e = readlink_path(&f, buf, PATH_MAX) < 0 ? errno : 0; 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2271:52: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2274:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.118 INFO:tasks.workunit.client.0.smithi039.stderr: 2274 | printf("%d/%d: readlink %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2274:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘rename_f’: 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2296:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr: 2296 | printf("%d/%d: rename - no filename\n", procid, opno); 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2296:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2311:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr: 2311 | printf("%d/%d: rename - no filename from %s\n", 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2311:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2318:42: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.119 INFO:tasks.workunit.client.0.smithi039.stderr: 2318 | e = rename_path(&f, &newf) < 0 ? errno : 0; 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2318:42: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2329:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: 2329 | printf("%d/%d: rename %s to %s %d\n", procid, opno, f.path, 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2329:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘rmdir_f’: 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2399:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: 2399 | printf("%d/%d: rmdir - no directory\n", procid, opno); 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2399:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2403:34: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: 2403 | e = rmdir_path(&f) < 0 ? errno : 0; 2024-02-19T19:11:51.120 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2403:34: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2408:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr: 2408 | printf("%d/%d: rmdir %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2408:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘stat_f’: 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2417:25: error: storage size of ‘stb’ isn’t known 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr: 2417 | struct stat64 stb; 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2423:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr: 2423 | printf("%d/%d: stat - no entries\n", procid, opno); 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2423:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.121 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2427:42: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: 2427 | e = lstat64_path(&f, &stb) < 0 ? errno : 0; 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2427:42: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2430:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: 2430 | printf("%d/%d: stat %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2430:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘symlink_f’: 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2458:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: 2458 | printf("%d/%d: symlink - no filename from %s\n", 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2458:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2464:32: error: ‘PATH_MAX’ undeclared (first use in this function) 2024-02-19T19:11:51.122 INFO:tasks.workunit.client.0.smithi039.stderr: 2464 | len = (int)(random() % PATH_MAX); 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2465:15: warning: incompatible implicit declaration of built-in function ‘malloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: 2465 | val = malloc(len + 1); 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2465:15: note: include ‘’ or provide a declaration of ‘malloc’ 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2467:17: warning: incompatible implicit declaration of built-in function ‘memset’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: 2467 | memset(val, 'x', len); 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2467:17: note: include ‘’ or provide a declaration of ‘memset’ 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2471:41: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: 2471 | e = symlink_path(val, &f) < 0 ? errno : 0; 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2471:41: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.123 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2475:9: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: 2475 | free(val); 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2475:9: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2477:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: 2477 | printf("%d/%d: symlink %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2477:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘sync_f’: 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2485:9: warning: implicit declaration of function ‘sync’; did you mean ‘sync_f’? [-Wimplicit-function-declaration] 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: 2485 | sync(); 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr: | sync_f 2024-02-19T19:11:51.124 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2487:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: 2487 | printf("%d/%d: sync\n", procid, opno); 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2487:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘truncate_f’: 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2495:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: 2495 | __int64_t lr; 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2496:9: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: 2496 | off64_t off; 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2497:25: error: storage size of ‘stb’ isn’t known 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: 2497 | struct stat64 stb; 2024-02-19T19:11:51.125 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2503:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: 2503 | printf("%d/%d: truncate - no filename\n", procid, opno); 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2503:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2507:41: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: 2507 | e = stat64_path(&f, &stb) < 0 ? errno : 0; 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2507:41: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2511:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: 2511 | printf("%d/%d: truncate - stat64 %s failed %d\n", 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2511:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2516:16: error: ‘__int64_t’ undeclared (first use in this function); did you mean ‘__int128_t’? 2024-02-19T19:11:51.126 INFO:tasks.workunit.client.0.smithi039.stderr: 2516 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2516:26: error: expected ‘)’ before ‘random’ 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: 2516 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | ~ ^~~~~~ 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | ) 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2517:16: error: ‘off64_t’ undeclared (first use in this function) 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: 2517 | off = (off64_t)(lr % MIN(stb.st_size + (1024 * 1024), MAXFSIZE)); 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2519:13: warning: implicit declaration of function ‘truncate64_path’ [-Wimplicit-function-declaration] 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: 2519 | e = truncate64_path(&f, off) < 0 ? errno : 0; 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~ 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2522:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: 2522 | printf("%d/%d: truncate %s %lld %d\n", procid, opno, f.path, 2024-02-19T19:11:51.127 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2522:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘unlink_f’: 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2539:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr: 2539 | printf("%d/%d: unlink - no file\n", procid, opno); 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2539:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2543:35: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr: 2543 | e = unlink_path(&f) < 0 ? errno : 0; 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2543:35: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2548:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr: 2548 | printf("%d/%d: unlink %s %d\n", procid, opno, f.path, e); 2024-02-19T19:11:51.128 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2548:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c: In function ‘write_f’: 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2612:9: error: unknown type name ‘size_t’ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: 2612 | size_t len; 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2612:9: note: ‘size_t’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2613:9: error: unknown type name ‘__int64_t’; did you mean ‘__int128_t’? 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: 2613 | __int64_t lr; 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2614:9: error: unknown type name ‘off64_t’ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: 2614 | off64_t off; 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.129 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2615:25: error: storage size of ‘stb’ isn’t known 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: 2615 | struct stat64 stb; 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~ 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2621:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: 2621 | printf("%d/%d: write - no filename\n", procid, opno); 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2621:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2625:28: error: ‘O_WRONLY’ undeclared (first use in this function) 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: 2625 | fd = open_path(&f, O_WRONLY); 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2626:22: error: ‘errno’ undeclared (first use in this function) 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: 2626 | e = fd < 0 ? errno : 0; 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~ 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2626:22: note: ‘errno’ is defined in header ‘’; did you forget to ‘#include ’? 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2630:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.130 INFO:tasks.workunit.client.0.smithi039.stderr: 2630 | printf("%d/%d: write - open %s failed %d\n", 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2630:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2637:25: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: 2637 | printf("%d/%d: write - fstat64 %s failed %d\n", 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2637:25: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2643:16: error: ‘__int64_t’ undeclared (first use in this function); did you mean ‘__int128_t’? 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: 2643 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: | __int128_t 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2643:26: error: expected ‘)’ before ‘random’ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: 2643 | lr = ((__int64_t)random() << 32) + random(); 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: | ~ ^~~~~~ 2024-02-19T19:11:51.131 INFO:tasks.workunit.client.0.smithi039.stderr: | ) 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2644:16: error: ‘off64_t’ undeclared (first use in this function) 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: 2644 | off = (off64_t)(lr % MIN(stb.st_size + (1024 * 1024), MAXFSIZE)); 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~ 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2646:26: error: ‘SEEK_SET’ undeclared (first use in this function) 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: 2646 | lseek64(fd, off, SEEK_SET); 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2648:15: warning: incompatible implicit declaration of built-in function ‘malloc’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: 2648 | buf = malloc(len); 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2648:15: note: include ‘’ or provide a declaration of ‘malloc’ 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2649:9: warning: incompatible implicit declaration of built-in function ‘memset’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: 2649 | memset(buf, nameseq & 0xff, len); 2024-02-19T19:11:51.132 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2649:9: note: include ‘’ or provide a declaration of ‘memset’ 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2651:9: warning: incompatible implicit declaration of built-in function ‘free’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr: 2651 | free(buf); 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2651:9: note: include ‘’ or provide a declaration of ‘free’ 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2653:17: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch] 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr: 2653 | printf("%d/%d: write %s [%lld,%ld] %d\n", 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~ 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:fsstress.c:2653:17: note: include ‘’ or provide a declaration of ‘printf’ 2024-02-19T19:11:51.133 INFO:tasks.workunit.client.0.smithi039.stderr:make: *** [: fsstress] Error 1 2024-02-19T19:11:51.133 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-19T19:11:51.134 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-19T19:11:51.440 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi039 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2024-02-19T19:11:51.647 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c97b0cb4c59b48a4866b8a2add1b65d9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi039 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2024-02-19T19:11:51.651 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-19T19:11:51.663 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-19T19:11:51.663 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T19:11:51.663 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:11:51.686 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-02-19T19:11:51.686 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:11:51.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:11:51.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 -- 172.21.15.39:0/976223521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 msgr2=0x7fdfa00ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/976223521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 0x7fdfa00ff6a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009880 tx=0x7fdf9002bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 -- 172.21.15.39:0/976223521 shutdown_connections 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/976223521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa0103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/976223521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 0x7fdfa00ff6a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 -- 172.21.15.39:0/976223521 >> 172.21.15.39:0/976223521 conn(0x7fdfa0075bb0 msgr2=0x7fdfa0075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.908+0000 7fdfa5c15640 1 -- 172.21.15.39:0/976223521 shutdown_connections 2024-02-19T19:11:51.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.909+0000 7fdfa5c15640 1 -- 172.21.15.39:0/976223521 wait complete. 2024-02-19T19:11:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.909+0000 7fdfa5c15640 1 Processor -- start 2024-02-19T19:11:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.909+0000 7fdfa5c15640 1 -- start start 2024-02-19T19:11:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdfa5c15640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 0x7fdfa010b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdfa5c15640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa010b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdfa5c15640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa010be50 con 0x7fdfa00ffc70 2024-02-19T19:11:51.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdfa5c15640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa010bfc0 con 0x7fdfa00ff2c0 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdf9effd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa010b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdf9effd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa010b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56580/0 (socket says 172.21.15.39:56580) 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdf9effd640 1 -- 172.21.15.39:0/733469126 learned_addr learned my addr 172.21.15.39:0/733469126 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdf9effd640 1 -- 172.21.15.39:0/733469126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 msgr2=0x7fdfa010b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdf9effd640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 0x7fdfa010b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.910+0000 7fdf9effd640 1 -- 172.21.15.39:0/733469126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf90009560 con 0x7fdfa00ffc70 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.911+0000 7fdf9effd640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa010b910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400e970 tx=0x7fdf9400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.911+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf9400cc90 con 0x7fdfa00ffc70 2024-02-19T19:11:51.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.911+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdf9400cdf0 con 0x7fdfa00ffc70 2024-02-19T19:11:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.911+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf94010430 con 0x7fdfa00ffc70 2024-02-19T19:11:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.911+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa010c240 con 0x7fdfa00ffc70 2024-02-19T19:11:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.911+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa01a2300 con 0x7fdfa00ffc70 2024-02-19T19:11:51.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.912+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa004e040 con 0x7fdfa00ffc70 2024-02-19T19:11:51.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.913+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fdf940046c0 con 0x7fdfa00ffc70 2024-02-19T19:11:51.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.914+0000 7fdf9cff9640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fdf8806d810 0x7fdf8806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:51.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.914+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fdf94014070 con 0x7fdfa00ffc70 2024-02-19T19:11:51.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.914+0000 7fdf9f7fe640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fdf8806d810 0x7fdf8806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:51.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.915+0000 7fdf9f7fe640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fdf8806d810 0x7fdf8806fcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf90036040 tx=0x7fdf90000bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:51.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:51.921+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf94057060 con 0x7fdfa00ffc70 2024-02-19T19:11:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.210+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfa010c180 con 0x7fdfa00ffc70 2024-02-19T19:11:52.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.212+0000 7fdf9cff9640 1 -- 172.21.15.39:0/733469126 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16853 (secure 0 0 0) 0x7fdf9405a720 con 0x7fdfa00ffc70 2024-02-19T19:11:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:52.236 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":85,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","created":"2024-02-19T18:54:43.947412+0000","modified":"2024-02-19T19:11:33.833874+0000","last_up_change":"2024-02-19T19:10:17.396873+0000","last_in_change":"2024-02-19T19:00:23.810529+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:59:26.986411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-19T19:01:57.224279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-19T19:01:58.037930+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91f217d9-d1a8-4658-b693-873f74d42408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6803","nonce":558256553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6805","nonce":558256553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6809","nonce":558256553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6807","nonce":558256553}]},"public_addr":"172.21.15.39:6803/558256553","cluster_addr":"172.21.15.39:6805/558256553","heartbeat_back_addr":"172.21.15.39:6809/558256553","heartbeat_front_addr":"172.21.15.39:6807/558256553","state":["exists","up"]},{"osd":1,"uuid":"4b6143b3-8bce-4be7-a199-57711f652a9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6811","nonce":4048799663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6813","nonce":4048799663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6817","nonce":4048799663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6815","nonce":4048799663}]},"public_addr":"172.21.15.39:6811/4048799663","cluster_addr":"172.21.15.39:6813/4048799663","heartbeat_back_addr":"172.21.15.39:6817/4048799663","heartbeat_front_addr":"172.21.15.39:6815/4048799663","state":["exists","up"]},{"osd":2,"uuid":"bcb75e1b-3f22-4e89-9bdd-ce8e833a3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6819","nonce":2401360837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6821","nonce":2401360837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6825","nonce":2401360837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6823","nonce":2401360837}]},"public_addr":"172.21.15.39:6819/2401360837","cluster_addr":"172.21.15.39:6821/2401360837","heartbeat_back_addr":"172.21.15.39:6825/2401360837","heartbeat_front_addr":"172.21.15.39:6823/2401360837","state":["exists","up"]},{"osd":3,"uuid":"8c3353e0-09f3-4582-af41-33ed9e1581c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6801","nonce":2382739218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6803","nonce":2382739218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6807","nonce":2382739218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6805","nonce":2382739218}]},"public_addr":"172.21.15.86:6801/2382739218","cluster_addr":"172.21.15.86:6803/2382739218","heartbeat_back_addr":"172.21.15.86:6807/2382739218","heartbeat_front_addr":"172.21.15.86:6805/2382739218","state":["exists","up"]},{"osd":4,"uuid":"62e52e70-7bbd-4395-98b1-2bc8eabc2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6809","nonce":560609481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6811","nonce":560609481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6815","nonce":560609481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6813","nonce":560609481}]},"public_addr":"172.21.15.86:6809/560609481","cluster_addr":"172.21.15.86:6811/560609481","heartbeat_back_addr":"172.21.15.86:6815/560609481","heartbeat_front_addr":"172.21.15.86:6813/560609481","state":["exists","up"]},{"osd":5,"uuid":"772c2485-d3a7-4ee6-9f6d-4f9535db5861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6817","nonce":3455419941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6819","nonce":3455419941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6823","nonce":3455419941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6821","nonce":3455419941}]},"public_addr":"172.21.15.86:6817/3455419941","cluster_addr":"172.21.15.86:6819/3455419941","heartbeat_back_addr":"172.21.15.86:6823/3455419941","heartbeat_front_addr":"172.21.15.86:6821/3455419941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:24.714559+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:54.443434+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:24.712567+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:49.208459+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:15.215844+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:41.317444+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6826/978975893":"2024-02-20T19:11:33.833848+0000","172.21.15.86:6825/2684130153":"2024-02-20T19:11:07.184426+0000","172.21.15.86:6824/2684130153":"2024-02-20T19:11:07.184426+0000","172.21.15.86:0/3588485196":"2024-02-20T19:05:27.023863+0000","172.21.15.86:6828/1232386891":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/3933707157":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/2610970659":"2024-02-20T19:06:13.407336+0000","172.21.15.39:6800/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1444159856":"2024-02-20T18:55:10.759262+0000","172.21.15.86:0/4117360583":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/4293838126":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/2615309126":"2024-02-20T19:04:44.155895+0000","172.21.15.86:0/1239352357":"2024-02-20T19:05:27.023863+0000","172.21.15.39:6800/2307886882":"2024-02-20T19:04:44.155895+0000","172.21.15.86:6827/978975893":"2024-02-20T19:11:33.833848+0000","172.21.15.39:0/2619661656":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/883565860":"2024-02-20T19:04:44.155895+0000","172.21.15.39:6827/2541159330":"2024-02-20T19:02:13.966534+0000","172.21.15.39:6801/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6801/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6801/913229036":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/1749005598":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6800/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.86:6829/1232386891":"2024-02-20T19:05:27.023863+0000","172.21.15.39:6801/2307886882":"2024-02-20T19:04:44.155895+0000","172.21.15.39:0/1614648654":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6800/913229036":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/2313120383":"2024-02-20T18:55:43.704412+0000","172.21.15.86:0/4112182782":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/3613879910":"2024-02-20T18:55:10.759262+0000","172.21.15.86:0/1077741088":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/2871115404":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/2362951092":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/3788984480":"2024-02-20T19:04:44.155895+0000","172.21.15.39:0/3009513678":"2024-02-20T19:04:44.155895+0000","172.21.15.39:6800/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6826/2541159330":"2024-02-20T19:02:13.966534+0000","172.21.15.39:0/1261064923":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/1410355349":"2024-02-20T19:06:13.407336+0000","172.21.15.86:0/4267507691":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/3608355132":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/2027083659":"2024-02-20T19:06:13.407336+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:11:52.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fdf8806d810 msgr2=0x7fdf8806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fdf8806d810 0x7fdf8806fcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf90036040 tx=0x7fdf90000bc0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 msgr2=0x7fdfa010b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa010b910 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400e970 tx=0x7fdf9400ed70 comp rx=0 tx=0).stop 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 shutdown_connections 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fdf8806d810 0x7fdf8806fcd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfa00ffc70 0x7fdfa010b910 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 --2- 172.21.15.39:0/733469126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa00ff2c0 0x7fdfa010b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.215+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 >> 172.21.15.39:0/733469126 conn(0x7fdfa0075bb0 msgr2=0x7fdfa00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:52.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.216+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 shutdown_connections 2024-02-19T19:11:52.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.216+0000 7fdfa5c15640 1 -- 172.21.15.39:0/733469126 wait complete. 2024-02-19T19:11:52.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:59:26.986411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-19T19:01:57.224279+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-19T19:01:58.037930+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T19:11:52.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T19:11:52.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 -- 172.21.15.39:0/919532596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 msgr2=0x7fc0f00ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 --2- 172.21.15.39:0/919532596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f00ff470 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009880 tx=0x7fc0e002bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 -- 172.21.15.39:0/919532596 shutdown_connections 2024-02-19T19:11:52.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 --2- 172.21.15.39:0/919532596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f00ff470 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 --2- 172.21.15.39:0/919532596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f0104fc0 0x7fc0f01053a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 -- 172.21.15.39:0/919532596 >> 172.21.15.39:0/919532596 conn(0x7fc0f0075b50 msgr2=0x7fc0f0075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 -- 172.21.15.39:0/919532596 shutdown_connections 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.341+0000 7fc0f8621640 1 -- 172.21.15.39:0/919532596 wait complete. 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f8621640 1 Processor -- start 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f8621640 1 -- start start 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f8621640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f0196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:52.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f8621640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f0104fc0 0x7fc0f01970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f8621640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01976a0 con 0x7fc0f00ff030 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f8621640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0197810 con 0x7fc0f0104fc0 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f63bd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f0196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f63bd640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f0196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56586/0 (socket says 172.21.15.39:56586) 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f63bd640 1 -- 172.21.15.39:0/2009579531 learned_addr learned my addr 172.21.15.39:0/2009579531 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.342+0000 7fc0f5bbc640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f0104fc0 0x7fc0f01970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0f63bd640 1 -- 172.21.15.39:0/2009579531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f0104fc0 msgr2=0x7fc0f01970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0f63bd640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f0104fc0 0x7fc0f01970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0f63bd640 1 -- 172.21.15.39:0/2009579531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0009560 con 0x7fc0f00ff030 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0f63bd640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f0196b90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00e970 tx=0x7fc0ec00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0ec00cc90 con 0x7fc0f00ff030 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc0ec00cdf0 con 0x7fc0f00ff030 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0ec010430 con 0x7fc0f00ff030 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f019fc00 con 0x7fc0f00ff030 2024-02-19T19:11:52.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.343+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f0194980 con 0x7fc0f00ff030 2024-02-19T19:11:52.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.344+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f004e040 con 0x7fc0f00ff030 2024-02-19T19:11:52.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.345+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fc0ec0046c0 con 0x7fc0f00ff030 2024-02-19T19:11:52.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.345+0000 7fc0e77fe640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc0dc06d810 0x7fc0dc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:52.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.345+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc0ec014070 con 0x7fc0f00ff030 2024-02-19T19:11:52.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.345+0000 7fc0f5bbc640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc0dc06d810 0x7fc0dc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:52.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.346+0000 7fc0f5bbc640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc0dc06d810 0x7fc0dc06fcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0006010 tx=0x7fc0e0005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:52.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.350+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0ec057140 con 0x7fc0f00ff030 2024-02-19T19:11:52.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.576+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0f010c1c0 con 0x7fc0f00ff030 2024-02-19T19:11:52.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.577+0000 7fc0e77fe640 1 -- 172.21.15.39:0/2009579531 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0ec05a800 con 0x7fc0f00ff030 2024-02-19T19:11:52.600 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-02-19T19:11:52.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc0dc06d810 msgr2=0x7fc0dc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc0dc06d810 0x7fc0dc06fcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0006010 tx=0x7fc0e0005dd0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 msgr2=0x7fc0f0196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f0196b90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00e970 tx=0x7fc0ec00ed70 comp rx=0 tx=0).stop 2024-02-19T19:11:52.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 shutdown_connections 2024-02-19T19:11:52.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fc0dc06d810 0x7fc0dc06fcd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0f0104fc0 0x7fc0f01970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 --2- 172.21.15.39:0/2009579531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0f00ff030 0x7fc0f0196b90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 >> 172.21.15.39:0/2009579531 conn(0x7fc0f0075b50 msgr2=0x7fc0f00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:52.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 shutdown_connections 2024-02-19T19:11:52.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.579+0000 7fc0f8621640 1 -- 172.21.15.39:0/2009579531 wait complete. 2024-02-19T19:11:52.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-19T19:11:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 -- 172.21.15.39:0/519074281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067340 msgr2=0x7f0e74067720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/519074281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067340 0x7f0e74067720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009880 tx=0x7f0e6402bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 -- 172.21.15.39:0/519074281 shutdown_connections 2024-02-19T19:11:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/519074281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e74067c60 0x7f0e7410c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/519074281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067340 0x7f0e74067720 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 -- 172.21.15.39:0/519074281 >> 172.21.15.39:0/519074281 conn(0x7f0e74075b50 msgr2=0x7f0e74075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:52.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 -- 172.21.15.39:0/519074281 shutdown_connections 2024-02-19T19:11:52.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.722+0000 7f0e78e4a640 1 -- 172.21.15.39:0/519074281 wait complete. 2024-02-19T19:11:52.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.723+0000 7f0e78e4a640 1 Processor -- start 2024-02-19T19:11:52.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.723+0000 7f0e78e4a640 1 -- start start 2024-02-19T19:11:52.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.723+0000 7f0e78e4a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e74067340 0x7f0e74197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:52.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.723+0000 7f0e78e4a640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 0x7f0e741983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:52.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.723+0000 7f0e78e4a640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e74198a80 con 0x7f0e74067c60 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.723+0000 7f0e78e4a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7419c7e0 con 0x7f0e74067340 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 0x7f0e741983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 0x7f0e741983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56594/0 (socket says 172.21.15.39:56594) 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 -- 172.21.15.39:0/2156246213 learned_addr learned my addr 172.21.15.39:0/2156246213 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e7259c640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e74067340 0x7f0e74197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 -- 172.21.15.39:0/2156246213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e74067340 msgr2=0x7f0e74197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e74067340 0x7f0e74197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 -- 172.21.15.39:0/2156246213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e64009560 con 0x7f0e74067c60 2024-02-19T19:11:52.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e69d9b640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 0x7f0e741983f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00e970 tx=0x7f0e5c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:52.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.724+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e5c00cc90 con 0x7f0e74067c60 2024-02-19T19:11:52.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.725+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0e5c00cdf0 con 0x7f0e74067c60 2024-02-19T19:11:52.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.725+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e5c018810 con 0x7f0e74067c60 2024-02-19T19:11:52.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.725+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7419cac0 con 0x7f0e74067c60 2024-02-19T19:11:52.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.725+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7419d020 con 0x7f0e74067c60 2024-02-19T19:11:52.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.726+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7404e040 con 0x7f0e74067c60 2024-02-19T19:11:52.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.726+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f0e5c010c00 con 0x7f0e74067c60 2024-02-19T19:11:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.727+0000 7f0e6bfff640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e6006d810 0x7f0e6006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.727+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f0e5c014070 con 0x7f0e74067c60 2024-02-19T19:11:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.727+0000 7f0e7259c640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e6006d810 0x7f0e6006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.728+0000 7f0e7259c640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e6006d810 0x7f0e6006fcd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0e64006010 tx=0x7f0e64005dd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:52.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.734+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e5c052d20 con 0x7f0e74067c60 2024-02-19T19:11:52.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:52 smithi039 ceph-mon[92224]: pgmap v208: 65 pgs: 65 active+clean; 124 MiB data, 642 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 6.7 MiB/s wr, 858 op/s 2024-02-19T19:11:52.833 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:52 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/733469126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:11:52.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.966+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f0e74199200 con 0x7f0e74067c60 2024-02-19T19:11:52.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.967+0000 7f0e6bfff640 1 -- 172.21.15.39:0/2156246213 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0e5c05ad90 con 0x7f0e74067c60 2024-02-19T19:11:52.990 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-02-19T19:11:52.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e6006d810 msgr2=0x7f0e6006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e6006d810 0x7f0e6006fcd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0e64006010 tx=0x7f0e64005dd0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 msgr2=0x7f0e741983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:52.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 0x7f0e741983f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00e970 tx=0x7f0e5c00ee30 comp rx=0 tx=0).stop 2024-02-19T19:11:52.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 shutdown_connections 2024-02-19T19:11:52.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f0e6006d810 0x7f0e6006fcd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e74067c60 0x7f0e741983f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 --2- 172.21.15.39:0/2156246213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e74067340 0x7f0e74197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:52.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 >> 172.21.15.39:0/2156246213 conn(0x7f0e74075b50 msgr2=0x7f0e740fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:52.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 shutdown_connections 2024-02-19T19:11:52.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:52.969+0000 7f0e78e4a640 1 -- 172.21.15.39:0/2156246213 wait complete. 2024-02-19T19:11:53.003 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-19T19:11:53.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:52 smithi086 ceph-mon[82675]: pgmap v208: 65 pgs: 65 active+clean; 124 MiB data, 642 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 6.7 MiB/s wr, 858 op/s 2024-02-19T19:11:53.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:52 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/733469126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:11:53.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/1828827394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e041060c0 msgr2=0x7f8e041064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/1828827394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e041060c0 0x7f8e041064a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009880 tx=0x7f8e0002bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/1828827394 shutdown_connections 2024-02-19T19:11:53.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/1828827394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e04067340 0x7f8e04067780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/1828827394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e041060c0 0x7f8e041064a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/1828827394 >> 172.21.15.39:0/1828827394 conn(0x7f8e04075b50 msgr2=0x7f8e04075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/1828827394 shutdown_connections 2024-02-19T19:11:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.086+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/1828827394 wait complete. 2024-02-19T19:11:53.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e0bd2a640 1 Processor -- start 2024-02-19T19:11:53.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e0bd2a640 1 -- start start 2024-02-19T19:11:53.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e0bd2a640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 0x7f8e040ff100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:53.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e0bd2a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e041060c0 0x7f8e040ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:53.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e0bd2a640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e040ffb80 con 0x7f8e04067340 2024-02-19T19:11:53.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e0bd2a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e040ffcf0 con 0x7f8e041060c0 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e09ac6640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 0x7f8e040ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e09ac6640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 0x7f8e040ff100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56618/0 (socket says 172.21.15.39:56618) 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.087+0000 7f8e09ac6640 1 -- 172.21.15.39:0/4232291773 learned_addr learned my addr 172.21.15.39:0/4232291773 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e092c5640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e041060c0 0x7f8e040ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e09ac6640 1 -- 172.21.15.39:0/4232291773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e041060c0 msgr2=0x7f8e040ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e09ac6640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e041060c0 0x7f8e040ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e09ac6640 1 -- 172.21.15.39:0/4232291773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00009560 con 0x7f8e04067340 2024-02-19T19:11:53.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e09ac6640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 0x7f8e040ff100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8e00000c00 tx=0x7f8e0002c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e00004420 con 0x7f8e04067340 2024-02-19T19:11:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e040fff10 con 0x7f8e04067340 2024-02-19T19:11:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e000054b0 con 0x7f8e04067340 2024-02-19T19:11:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e00034440 con 0x7f8e04067340 2024-02-19T19:11:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.088+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e041a2210 con 0x7f8e04067340 2024-02-19T19:11:53.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.089+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0404e040 con 0x7f8e04067340 2024-02-19T19:11:53.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.091+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f8e00004f20 con 0x7f8e04067340 2024-02-19T19:11:53.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.091+0000 7f8dfaffd640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8df006d810 0x7f8df006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:53.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.091+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8e000afdd0 con 0x7f8e04067340 2024-02-19T19:11:53.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.091+0000 7f8e092c5640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8df006d810 0x7f8df006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:53.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.092+0000 7f8e092c5640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8df006d810 0x7f8df006fcd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8e041008f0 tx=0x7f8df400b590 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:53.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.099+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e00079650 con 0x7f8e04067340 2024-02-19T19:11:53.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.286+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f8e04108520 con 0x7f8e04067340 2024-02-19T19:11:53.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.287+0000 7f8dfaffd640 1 -- 172.21.15.39:0/4232291773 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8e00041020 con 0x7f8e04067340 2024-02-19T19:11:53.310 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-02-19T19:11:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8df006d810 msgr2=0x7f8df006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8df006d810 0x7f8df006fcd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8e041008f0 tx=0x7f8df400b590 comp rx=0 tx=0).stop 2024-02-19T19:11:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 msgr2=0x7f8e040ff100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 0x7f8e040ff100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8e00000c00 tx=0x7f8e0002c6f0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 shutdown_connections 2024-02-19T19:11:53.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f8df006d810 0x7f8df006fcd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e041060c0 0x7f8e040ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 --2- 172.21.15.39:0/4232291773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e04067340 0x7f8e040ff100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 >> 172.21.15.39:0/4232291773 conn(0x7f8e04075b50 msgr2=0x7f8e040fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:53.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 shutdown_connections 2024-02-19T19:11:53.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.289+0000 7f8e0bd2a640 1 -- 172.21.15.39:0/4232291773 wait complete. 2024-02-19T19:11:53.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T19:11:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.408+0000 7fa6c885c640 1 -- 172.21.15.39:0/2574854120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 msgr2=0x7fa6c40ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.408+0000 7fa6c885c640 1 --2- 172.21.15.39:0/2574854120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c40ff6a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009880 tx=0x7fa6b402bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 -- 172.21.15.39:0/2574854120 shutdown_connections 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 --2- 172.21.15.39:0/2574854120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6c40ffc70 0x7fa6c4103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 --2- 172.21.15.39:0/2574854120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c40ff6a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 -- 172.21.15.39:0/2574854120 >> 172.21.15.39:0/2574854120 conn(0x7fa6c4075bb0 msgr2=0x7fa6c4075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 -- 172.21.15.39:0/2574854120 shutdown_connections 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 -- 172.21.15.39:0/2574854120 wait complete. 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 Processor -- start 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.409+0000 7fa6c885c640 1 -- start start 2024-02-19T19:11:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c885c640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c4195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c885c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6c40ffc70 0x7fa6c41962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c885c640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4196940 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c885c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c419a710 con 0x7fa6c40ffc70 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c4195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c4195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56640/0 (socket says 172.21.15.39:56640) 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 -- 172.21.15.39:0/3561755171 learned_addr learned my addr 172.21.15.39:0/3561755171 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6bbfff640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6c40ffc70 0x7fa6c41962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:53.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 -- 172.21.15.39:0/3561755171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6c40ffc70 msgr2=0x7fa6c41962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6c40ffc70 0x7fa6c41962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 -- 172.21.15.39:0/3561755171 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4009560 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6c259c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c4195d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009230 tx=0x7fa6b4004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.410+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b40044a0 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.411+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa6b40054b0 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.411+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b4034440 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.411+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c419a990 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.411+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c419ae90 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.412+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c404e040 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.413+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fa6b4003780 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.413+0000 7fa6bb7fe640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa6b006d810 0x7fa6b006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:53.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.413+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa6b40afa20 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.413+0000 7fa6bbfff640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa6b006d810 0x7fa6b006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:53.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.414+0000 7fa6bbfff640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa6b006d810 0x7fa6b006fcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4197380 tx=0x7fa6ac00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:53.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.418+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa6b4079230 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.619+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa6c410c270 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.620+0000 7fa6bb7fe640 1 -- 172.21.15.39:0/3561755171 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 43 entries v85) v1 ==== 81+0+2541 (secure 0 0 0) 0x7fa6b407c8f0 con 0x7fa6c40ff2c0 2024-02-19T19:11:53.643 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6826/978975893 2024-02-20T19:11:33.833848+0000 2024-02-19T19:11:53.643 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6825/2684130153 2024-02-20T19:11:07.184426+0000 2024-02-19T19:11:53.643 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6824/2684130153 2024-02-20T19:11:07.184426+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/3588485196 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6828/1232386891 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3933707157 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2610970659 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/728525375 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1444159856 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/4117360583 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.644 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/4293838126 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2615309126 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/1239352357 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2307886882 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6827/978975893 2024-02-20T19:11:33.833848+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2619661656 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/883565860 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6827/2541159330 2024-02-20T19:02:13.966534+0000 2024-02-19T19:11:53.645 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/728525375 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2039816448 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2158365942 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/913229036 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1749005598 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2039816448 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6829/1232386891 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2307886882 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1614648654 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/913229036 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2313120383 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/4112182782 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3613879910 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/1077741088 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2871115404 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2362951092 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3788984480 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3009513678 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:53.647 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2158365942 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:53.648 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6826/2541159330 2024-02-20T19:02:13.966534+0000 2024-02-19T19:11:53.648 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1261064923 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:53.648 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1410355349 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.648 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/4267507691 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:53.648 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3608355132 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.648 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2027083659 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:53.649 INFO:teuthology.orchestra.run.smithi039.stderr:listed 43 entries 2024-02-19T19:11:53.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa6b006d810 msgr2=0x7fa6b006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa6b006d810 0x7fa6b006fcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4197380 tx=0x7fa6ac00b410 comp rx=0 tx=0).stop 2024-02-19T19:11:53.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 msgr2=0x7fa6c4195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c4195d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009230 tx=0x7fa6b4004f00 comp rx=0 tx=0).stop 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 shutdown_connections 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fa6b006d810 0x7fa6b006fcd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6c40ffc70 0x7fa6c41962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 --2- 172.21.15.39:0/3561755171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6c40ff2c0 0x7fa6c4195d70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 >> 172.21.15.39:0/3561755171 conn(0x7fa6c4075bb0 msgr2=0x7fa6c4103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.622+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 shutdown_connections 2024-02-19T19:11:53.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:53.623+0000 7fa6c885c640 1 -- 172.21.15.39:0/3561755171 wait complete. 2024-02-19T19:11:53.656 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-19T19:11:53.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:11:53.657 DEBUG:teuthology.orchestra.run.smithi039:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-19T19:11:53.684 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-02-19T19:11:53.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:11:53.685 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T19:11:53.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:53 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/2009579531' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:11:53.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:53 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/2156246213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:11:53.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:53 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/4232291773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:11:54.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:53 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/2009579531' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:11:54.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:53 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/2156246213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:11:54.037 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:53 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/4232291773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:11:54.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:54 smithi039 ceph-mon[92224]: pgmap v209: 65 pgs: 65 active+clean; 128 MiB data, 690 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 6.9 MiB/s wr, 896 op/s 2024-02-19T19:11:54.999 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:54 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/3561755171' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T19:11:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:54 smithi086 ceph-mon[82675]: pgmap v209: 65 pgs: 65 active+clean; 128 MiB data, 690 MiB used, 536 GiB / 536 GiB avail; 8.0 KiB/s rd, 6.9 MiB/s wr, 896 op/s 2024-02-19T19:11:55.036 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:54 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/3561755171' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T19:11:57.239 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T19:11:57.239 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:11:57.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: pgmap v210: 65 pgs: 65 active+clean; 143 MiB data, 736 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 7.3 MiB/s wr, 785 op/s 2024-02-19T19:11:57.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: Upgrade: Setting container_image for all mds 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.aygwnv"}]': finished 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.omsmsr"}]': finished 2024-02-19T19:11:57.251 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:11:57.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.euvbgw"}]': finished 2024-02-19T19:11:57.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:11:57.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.idtugy"}]': finished 2024-02-19T19:11:57.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-02-19T19:11:57.252 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:57 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-02-19T19:11:57.259 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-02-19T19:11:57.259 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:11:57.316 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:11:57.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.435+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/395403136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8100c40 msgr2=0x7f2cd8104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.435+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/395403136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8100c40 0x7f2cd8104c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40097e0 tx=0x7f2cc402bc30 comp rx=0 tx=0).stop 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.435+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/395403136 shutdown_connections 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.435+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/395403136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd8100c40 0x7f2cd8104c80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.435+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/395403136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.435+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/395403136 >> 172.21.15.39:0/395403136 conn(0x7f2cd80fbad0 msgr2=0x7f2cd80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.436+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/395403136 shutdown_connections 2024-02-19T19:11:57.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.436+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/395403136 wait complete. 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.437+0000 7f2cdf0b6640 1 Processor -- start 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.437+0000 7f2cdf0b6640 1 -- start start 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.437+0000 7f2cdf0b6640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.437+0000 7f2cdf0b6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 0x7f2cd819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cdf0b6640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81968a0 con 0x7f2cd8100290 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cdf0b6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8196a10 con 0x7f2cd81962a0 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cdce52640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:57.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cd7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 0x7f2cd819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cdce52640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56124/0 (socket says 172.21.15.39:56124) 2024-02-19T19:11:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cd7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 0x7f2cd819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:55216/0 (socket says 172.21.15.39:55216) 2024-02-19T19:11:57.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cdce52640 1 -- 172.21.15.39:0/324994492 learned_addr learned my addr 172.21.15.39:0/324994492 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cd7fff640 1 -- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 msgr2=0x7f2cd8195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cd7fff640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cd7fff640 1 -- 172.21.15.39:0/324994492 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4009490 con 0x7f2cd81962a0 2024-02-19T19:11:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.438+0000 7f2cdce52640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:11:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.439+0000 7f2cd7fff640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 0x7f2cd819a700 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4037040 tx=0x7f2cc4034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.439+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cc4034990 con 0x7f2cd81962a0 2024-02-19T19:11:57.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.439+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2cc402ca80 con 0x7f2cd81962a0 2024-02-19T19:11:57.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.439+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cc4032600 con 0x7f2cd81962a0 2024-02-19T19:11:57.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.439+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd819aca0 con 0x7f2cd81962a0 2024-02-19T19:11:57.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.440+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd819b140 con 0x7f2cd81962a0 2024-02-19T19:11:57.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.441+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f2cc403b040 con 0x7f2cd81962a0 2024-02-19T19:11:57.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.441+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd804e040 con 0x7f2cd81962a0 2024-02-19T19:11:57.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.442+0000 7f2cd5ffb640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2cc806d860 0x7f2cc806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:57.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.442+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2cc40af1a0 con 0x7f2cd81962a0 2024-02-19T19:11:57.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.442+0000 7f2cdce52640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2cc806d860 0x7f2cc806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:57.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.443+0000 7f2cdce52640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2cc806d860 0x7f2cc806fd20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0058c0 tx=0x7f2ccc009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:57.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.448+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2cc40789b0 con 0x7f2cd81962a0 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: pgmap v210: 65 pgs: 65 active+clean; 143 MiB data, 736 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s rd, 7.3 MiB/s wr, 785 op/s 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:11:57.478 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: Upgrade: Setting container_image for all mds 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.aygwnv"}]: dispatch 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.aygwnv"}]': finished 2024-02-19T19:11:57.479 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.omsmsr"}]: dispatch 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi039.omsmsr"}]': finished 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.euvbgw"}]: dispatch 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.euvbgw"}]': finished 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.idtugy"}]: dispatch 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi086.idtugy"}]': finished 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-02-19T19:11:57.480 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:57 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-02-19T19:11:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.732+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cd8108590 con 0x7f2cd81962a0 2024-02-19T19:11:57.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.733+0000 7f2cd5ffb640 1 -- 172.21.15.39:0/324994492 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16853 (secure 0 0 0) 0x7f2cc407c070 con 0x7f2cd81962a0 2024-02-19T19:11:57.756 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-02-19T19:11:57.756 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":85,"fsid":"1026b4de-cf58-11ee-95bb-87774f69a715","created":"2024-02-19T18:54:43.947412+0000","modified":"2024-02-19T19:11:33.833874+0000","last_up_change":"2024-02-19T19:10:17.396873+0000","last_in_change":"2024-02-19T19:00:23.810529+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:59:26.986411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-19T19:01:57.224279+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-19T19:01:58.037930+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91f217d9-d1a8-4658-b693-873f74d42408","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":78,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6803","nonce":558256553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6805","nonce":558256553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6809","nonce":558256553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":558256553},{"type":"v1","addr":"172.21.15.39:6807","nonce":558256553}]},"public_addr":"172.21.15.39:6803/558256553","cluster_addr":"172.21.15.39:6805/558256553","heartbeat_back_addr":"172.21.15.39:6809/558256553","heartbeat_front_addr":"172.21.15.39:6807/558256553","state":["exists","up"]},{"osd":1,"uuid":"4b6143b3-8bce-4be7-a199-57711f652a9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":58,"up_thru":78,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6811","nonce":4048799663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6813","nonce":4048799663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6817","nonce":4048799663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":4048799663},{"type":"v1","addr":"172.21.15.39:6815","nonce":4048799663}]},"public_addr":"172.21.15.39:6811/4048799663","cluster_addr":"172.21.15.39:6813/4048799663","heartbeat_back_addr":"172.21.15.39:6817/4048799663","heartbeat_front_addr":"172.21.15.39:6815/4048799663","state":["exists","up"]},{"osd":2,"uuid":"bcb75e1b-3f22-4e89-9bdd-ce8e833a3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":59,"up_from":63,"up_thru":78,"down_at":60,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6819","nonce":2401360837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6821","nonce":2401360837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6825","nonce":2401360837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2401360837},{"type":"v1","addr":"172.21.15.39:6823","nonce":2401360837}]},"public_addr":"172.21.15.39:6819/2401360837","cluster_addr":"172.21.15.39:6821/2401360837","heartbeat_back_addr":"172.21.15.39:6825/2401360837","heartbeat_front_addr":"172.21.15.39:6823/2401360837","state":["exists","up"]},{"osd":3,"uuid":"8c3353e0-09f3-4582-af41-33ed9e1581c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":64,"up_from":68,"up_thru":78,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6801","nonce":2382739218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6803","nonce":2382739218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6807","nonce":2382739218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":2382739218},{"type":"v1","addr":"172.21.15.86:6805","nonce":2382739218}]},"public_addr":"172.21.15.86:6801/2382739218","cluster_addr":"172.21.15.86:6803/2382739218","heartbeat_back_addr":"172.21.15.86:6807/2382739218","heartbeat_front_addr":"172.21.15.86:6805/2382739218","state":["exists","up"]},{"osd":4,"uuid":"62e52e70-7bbd-4395-98b1-2bc8eabc2fb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":69,"up_from":73,"up_thru":78,"down_at":70,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6809","nonce":560609481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6811","nonce":560609481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6815","nonce":560609481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":560609481},{"type":"v1","addr":"172.21.15.86:6813","nonce":560609481}]},"public_addr":"172.21.15.86:6809/560609481","cluster_addr":"172.21.15.86:6811/560609481","heartbeat_back_addr":"172.21.15.86:6815/560609481","heartbeat_front_addr":"172.21.15.86:6813/560609481","state":["exists","up"]},{"osd":5,"uuid":"772c2485-d3a7-4ee6-9f6d-4f9535db5861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":74,"up_from":78,"up_thru":78,"down_at":75,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6817","nonce":3455419941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6819","nonce":3455419941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6823","nonce":3455419941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3455419941},{"type":"v1","addr":"172.21.15.86:6821","nonce":3455419941}]},"public_addr":"172.21.15.86:6817/3455419941","cluster_addr":"172.21.15.86:6819/3455419941","heartbeat_back_addr":"172.21.15.86:6823/3455419941","heartbeat_front_addr":"172.21.15.86:6821/3455419941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:24.714559+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:54.443434+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:24.712567+0000","dead_epoch":59},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:59:49.208459+0000","dead_epoch":64},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:15.215844+0000","dead_epoch":69},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T19:00:41.317444+0000","dead_epoch":74}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6826/978975893":"2024-02-20T19:11:33.833848+0000","172.21.15.86:6825/2684130153":"2024-02-20T19:11:07.184426+0000","172.21.15.86:6824/2684130153":"2024-02-20T19:11:07.184426+0000","172.21.15.39:6801/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/2158365942":"2024-02-20T18:57:00.645468+0000","172.21.15.86:6828/1232386891":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/2610970659":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/3933707157":"2024-02-20T18:57:00.645468+0000","172.21.15.39:6800/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/1444159856":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/3608355132":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/4293838126":"2024-02-20T19:06:13.407336+0000","172.21.15.86:0/4117360583":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/2615309126":"2024-02-20T19:04:44.155895+0000","172.21.15.86:0/1239352357":"2024-02-20T19:05:27.023863+0000","172.21.15.39:6800/2307886882":"2024-02-20T19:04:44.155895+0000","172.21.15.39:0/883565860":"2024-02-20T19:04:44.155895+0000","172.21.15.39:6827/2541159330":"2024-02-20T19:02:13.966534+0000","172.21.15.39:6801/728525375":"2024-02-20T18:55:10.759262+0000","172.21.15.39:6801/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6800/2039816448":"2024-02-20T18:55:43.704412+0000","172.21.15.86:6829/1232386891":"2024-02-20T19:05:27.023863+0000","172.21.15.39:6801/2307886882":"2024-02-20T19:04:44.155895+0000","172.21.15.86:0/3588485196":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/1614648654":"2024-02-20T18:55:10.759262+0000","172.21.15.86:6827/978975893":"2024-02-20T19:11:33.833848+0000","172.21.15.39:0/2619661656":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/3613879910":"2024-02-20T18:55:10.759262+0000","172.21.15.39:0/2027083659":"2024-02-20T19:06:13.407336+0000","172.21.15.86:0/1077741088":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/2871115404":"2024-02-20T18:55:43.704412+0000","172.21.15.39:0/1410355349":"2024-02-20T19:06:13.407336+0000","172.21.15.39:0/1261064923":"2024-02-20T18:57:00.645468+0000","172.21.15.39:0/3009513678":"2024-02-20T19:04:44.155895+0000","172.21.15.86:0/4112182782":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/2313120383":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6800/913229036":"2024-02-20T19:06:13.407336+0000","172.21.15.39:6826/2541159330":"2024-02-20T19:02:13.966534+0000","172.21.15.39:0/3788984480":"2024-02-20T19:04:44.155895+0000","172.21.15.39:0/2362951092":"2024-02-20T19:06:13.407336+0000","172.21.15.86:0/4267507691":"2024-02-20T19:05:27.023863+0000","172.21.15.39:0/1749005598":"2024-02-20T18:55:43.704412+0000","172.21.15.39:6801/913229036":"2024-02-20T19:06:13.407336+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:11:57.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2cc806d860 msgr2=0x7f2cc806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:57.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2cc806d860 0x7f2cc806fd20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0058c0 tx=0x7f2ccc009260 comp rx=0 tx=0).stop 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 msgr2=0x7f2cd819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 0x7f2cd819a700 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4037040 tx=0x7f2cc4034440 comp rx=0 tx=0).stop 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 shutdown_connections 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f2cc806d860 0x7f2cc806fd20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd81962a0 0x7f2cd819a700 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4037040 tx=0x7f2cc4034440 comp rx=0 tx=0).stop 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 --2- 172.21.15.39:0/324994492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cd8100290 0x7f2cd8195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.735+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 >> 172.21.15.39:0/324994492 conn(0x7f2cd80fbad0 msgr2=0x7f2cd80fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.736+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 shutdown_connections 2024-02-19T19:11:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.736+0000 7f2cdf0b6640 1 -- 172.21.15.39:0/324994492 wait complete. 2024-02-19T19:11:57.774 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:59:26.986411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-19T19:01:57.224279+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-19T19:01:58.037930+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T19:11:57.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T19:11:57.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 -- 172.21.15.39:0/4266361235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780ff3d0 msgr2=0x7f8478106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:57.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 --2- 172.21.15.39:0/4266361235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780ff3d0 0x7f8478106ef0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f846c009880 tx=0x7f846c02bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 -- 172.21.15.39:0/4266361235 shutdown_connections 2024-02-19T19:11:57.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 --2- 172.21.15.39:0/4266361235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780ff3d0 0x7f8478106ef0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 --2- 172.21.15.39:0/4266361235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84780fea20 0x7f84780fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 -- 172.21.15.39:0/4266361235 >> 172.21.15.39:0/4266361235 conn(0x7f84780741f0 msgr2=0x7f8478074600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:57.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 -- 172.21.15.39:0/4266361235 shutdown_connections 2024-02-19T19:11:57.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.879+0000 7f847dda3640 1 -- 172.21.15.39:0/4266361235 wait complete. 2024-02-19T19:11:57.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.880+0000 7f847dda3640 1 Processor -- start 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.880+0000 7f847dda3640 1 -- start start 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f847dda3640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 0x7f8478197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f847dda3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8478198530 0x7f847819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f847dda3640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8478198b30 con 0x7f84780fea20 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f847dda3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8478198ca0 con 0x7f8478198530 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f84777fe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 0x7f8478197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f84777fe640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 0x7f8478197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56138/0 (socket says 172.21.15.39:56138) 2024-02-19T19:11:57.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f84777fe640 1 -- 172.21.15.39:0/1181445820 learned_addr learned my addr 172.21.15.39:0/1181445820 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f8476ffd640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8478198530 0x7f847819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f84777fe640 1 -- 172.21.15.39:0/1181445820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8478198530 msgr2=0x7f847819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f84777fe640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8478198530 0x7f847819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.881+0000 7f84777fe640 1 -- 172.21.15.39:0/1181445820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c009560 con 0x7f84780fea20 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.882+0000 7f84777fe640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 0x7f8478197ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f846800e9a0 tx=0x7f846800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.882+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f846800cd50 con 0x7f84780fea20 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.882+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f846800ceb0 con 0x7f84780fea20 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.882+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847819cf90 con 0x7f84780fea20 2024-02-19T19:11:57.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.882+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8468010660 con 0x7f84780fea20 2024-02-19T19:11:57.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.882+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847819d4f0 con 0x7f84780fea20 2024-02-19T19:11:57.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.883+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8464005280 con 0x7f84780fea20 2024-02-19T19:11:57.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.884+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f84680046c0 con 0x7f84780fea20 2024-02-19T19:11:57.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.885+0000 7f8474ff9640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f846006d810 0x7f846006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:57.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.885+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8468014070 con 0x7f84780fea20 2024-02-19T19:11:57.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.886+0000 7f8476ffd640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f846006d810 0x7f846006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:57.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.887+0000 7f8476ffd640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f846006d810 0x7f846006fcd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f846c000c00 tx=0x7f846c005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:57.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:57.894+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f846805abd0 con 0x7f84780fea20 2024-02-19T19:11:58.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.211+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8464005ab0 con 0x7f84780fea20 2024-02-19T19:11:58.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.211+0000 7f8474ff9640 1 -- 172.21.15.39:0/1181445820 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f846805a760 con 0x7f84780fea20 2024-02-19T19:11:58.234 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-02-19T19:11:58.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f846006d810 msgr2=0x7f846006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f846006d810 0x7f846006fcd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f846c000c00 tx=0x7f846c005fd0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 msgr2=0x7f8478197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 0x7f8478197ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f846800e9a0 tx=0x7f846800eda0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 shutdown_connections 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f846006d810 0x7f846006fcd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8478198530 0x7f847819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 --2- 172.21.15.39:0/1181445820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84780fea20 0x7f8478197ff0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 >> 172.21.15.39:0/1181445820 conn(0x7f84780741f0 msgr2=0x7f84780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:58.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 shutdown_connections 2024-02-19T19:11:58.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.214+0000 7f847dda3640 1 -- 172.21.15.39:0/1181445820 wait complete. 2024-02-19T19:11:58.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-19T19:11:58.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 -- 172.21.15.39:0/3361556569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68100c40 msgr2=0x7efc68104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/3361556569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68100c40 0x7efc68104c80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009880 tx=0x7efc5c02bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 -- 172.21.15.39:0/3361556569 shutdown_connections 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/3361556569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68100c40 0x7efc68104c80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/3361556569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc68100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 -- 172.21.15.39:0/3361556569 >> 172.21.15.39:0/3361556569 conn(0x7efc680fbb10 msgr2=0x7efc680fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 -- 172.21.15.39:0/3361556569 shutdown_connections 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.357+0000 7efc6d9d0640 1 -- 172.21.15.39:0/3361556569 wait complete. 2024-02-19T19:11:58.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc6d9d0640 1 Processor -- start 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc6d9d0640 1 -- start start 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc6d9d0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc680725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc6d9d0640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68072ae0 0x7efc68072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc6d9d0640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc681a1f00 con 0x7efc68072ae0 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc6d9d0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc681a2040 con 0x7efc68100290 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc66ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc680725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:58.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.358+0000 7efc667fc640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68072ae0 0x7efc68072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc66ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc680725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.39:55242/0 (socket says 172.21.15.39:55242) 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc66ffd640 1 -- 172.21.15.39:0/155817754 learned_addr learned my addr 172.21.15.39:0/155817754 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc66ffd640 1 -- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68072ae0 msgr2=0x7efc68072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc66ffd640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68072ae0 0x7efc68072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc66ffd640 1 -- 172.21.15.39:0/155817754 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c009560 con 0x7efc68100290 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc66ffd640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc680725a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efc5800e970 tx=0x7efc5800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5800cc90 con 0x7efc68100290 2024-02-19T19:11:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efc5800cdf0 con 0x7efc68100290 2024-02-19T19:11:58.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc681a2180 con 0x7efc68100290 2024-02-19T19:11:58.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc580187e0 con 0x7efc68100290 2024-02-19T19:11:58.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.359+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc681a26e0 con 0x7efc68100290 2024-02-19T19:11:58.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.361+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6804e040 con 0x7efc68100290 2024-02-19T19:11:58.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.361+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7efc58010c00 con 0x7efc68100290 2024-02-19T19:11:58.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.361+0000 7efc6c9ce640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc5006d810 0x7efc5006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:58.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.361+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7efc58014070 con 0x7efc68100290 2024-02-19T19:11:58.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.362+0000 7efc667fc640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc5006d810 0x7efc5006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:58.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.362+0000 7efc667fc640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc5006d810 0x7efc5006fcd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc5c000c00 tx=0x7efc5c005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.367+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc58052d20 con 0x7efc68100290 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 3 up:standby 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: Upgrade: Setting container_image for all rgw 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:58.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: Upgrade: Setting container_image for all rbd-mirror 2024-02-19T19:11:58.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:58.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:58.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:58 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/324994492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:11:58.463 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-02-19T19:11:58.463 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 3 up:standby 2024-02-19T19:11:58.463 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: fsmap cephfs:1 {0=cephfs.smithi039.omsmsr=up:active} 1 up:standby-replay 2 up:standby 2024-02-19T19:11:58.463 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:58.463 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: Upgrade: Setting container_image for all rgw 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: Upgrade: Setting container_image for all rbd-mirror 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:11:58.464 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:58 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/324994492' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:11:58.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.686+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7efc68108590 con 0x7efc68100290 2024-02-19T19:11:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.686+0000 7efc6c9ce640 1 -- 172.21.15.39:0/155817754 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7efc5805ace0 con 0x7efc68100290 2024-02-19T19:11:58.709 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc5006d810 msgr2=0x7efc5006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc5006d810 0x7efc5006fcd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc5c000c00 tx=0x7efc5c005fd0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 msgr2=0x7efc680725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc680725a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efc5800e970 tx=0x7efc5800ee30 comp rx=0 tx=0).stop 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 shutdown_connections 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7efc5006d810 0x7efc5006fcd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc68072ae0 0x7efc68072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 --2- 172.21.15.39:0/155817754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc68100290 0x7efc680725a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 >> 172.21.15.39:0/155817754 conn(0x7efc680fbb10 msgr2=0x7efc680fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 shutdown_connections 2024-02-19T19:11:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.689+0000 7efc6d9d0640 1 -- 172.21.15.39:0/155817754 wait complete. 2024-02-19T19:11:58.723 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-19T19:11:58.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.815+0000 7fb9942f7640 1 -- 172.21.15.39:0/3412022439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 msgr2=0x7fb98c1006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 --2- 172.21.15.39:0/3412022439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c1006b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb988009880 tx=0x7fb98802bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 -- 172.21.15.39:0/3412022439 shutdown_connections 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 --2- 172.21.15.39:0/3412022439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c100c80 0x7fb98c104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 --2- 172.21.15.39:0/3412022439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c1006b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 -- 172.21.15.39:0/3412022439 >> 172.21.15.39:0/3412022439 conn(0x7fb98c0fbb50 msgr2=0x7fb98c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 -- 172.21.15.39:0/3412022439 shutdown_connections 2024-02-19T19:11:58.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 -- 172.21.15.39:0/3412022439 wait complete. 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.816+0000 7fb9942f7640 1 Processor -- start 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb9942f7640 1 -- start start 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb9942f7640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb9942f7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c196300 0x7fb98c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb9942f7640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c196900 con 0x7fb98c1002d0 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb9942f7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c196a70 con 0x7fb98c196300 2024-02-19T19:11:58.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb992093640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb992093640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56196/0 (socket says 172.21.15.39:56196) 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb992093640 1 -- 172.21.15.39:0/4196015677 learned_addr learned my addr 172.21.15.39:0/4196015677 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb991892640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c196300 0x7fb98c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb991892640 1 -- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 msgr2=0x7fb98c195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb991892640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.817+0000 7fb991892640 1 -- 172.21.15.39:0/4196015677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988009560 con 0x7fb98c196300 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb992093640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb991892640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c196300 0x7fb98c19a760 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00e940 tx=0x7fb97c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb97c009620 con 0x7fb98c196300 2024-02-19T19:11:58.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98c19ad60 con 0x7fb98c196300 2024-02-19T19:11:58.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98c19b2c0 con 0x7fb98c196300 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb97c0044e0 con 0x7fb98c196300 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.818+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb97c0049d0 con 0x7fb98c196300 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.819+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb970005280 con 0x7fb98c196300 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.820+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7fb97c004030 con 0x7fb98c196300 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.820+0000 7fb9837fe640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb97806d860 0x7fb97806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.820+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fb97c014070 con 0x7fb98c196300 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.820+0000 7fb992093640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb97806d860 0x7fb97806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:58.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.821+0000 7fb992093640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb97806d860 0x7fb97806fd20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb988006010 tx=0x7fb98802cf90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:58.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:58.825+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb97c056e90 con 0x7fb98c196300 2024-02-19T19:11:59.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.140+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb970005ab0 con 0x7fb98c196300 2024-02-19T19:11:59.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.142+0000 7fb9837fe640 1 -- 172.21.15.39:0/4196015677 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v85) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb97c05a550 con 0x7fb98c196300 2024-02-19T19:11:59.164 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-02-19T19:11:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb97806d860 msgr2=0x7fb97806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb97806d860 0x7fb97806fd20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb988006010 tx=0x7fb98802cf90 comp rx=0 tx=0).stop 2024-02-19T19:11:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c196300 msgr2=0x7fb98c19a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c196300 0x7fb98c19a760 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00e940 tx=0x7fb97c00ee00 comp rx=0 tx=0).stop 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 shutdown_connections 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7fb97806d860 0x7fb97806fd20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98c196300 0x7fb98c19a760 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 --2- 172.21.15.39:0/4196015677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb98c1002d0 0x7fb98c195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 >> 172.21.15.39:0/4196015677 conn(0x7fb98c0fbb50 msgr2=0x7fb98c0fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 shutdown_connections 2024-02-19T19:11:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.144+0000 7fb9942f7640 1 -- 172.21.15.39:0/4196015677 wait complete. 2024-02-19T19:11:59.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-19T19:11:59.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/187204785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 msgr2=0x7f7a64067990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:59.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/187204785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64067990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009880 tx=0x7f7a6002bce0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/187204785 shutdown_connections 2024-02-19T19:11:59.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/187204785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64067f60 0x7f7a64104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/187204785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64067990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/187204785 >> 172.21.15.39:0/187204785 conn(0x7f7a64075bb0 msgr2=0x7f7a64075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:59.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/187204785 shutdown_connections 2024-02-19T19:11:59.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.304+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/187204785 wait complete. 2024-02-19T19:11:59.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.305+0000 7f7a6b7cb640 1 Processor -- start 2024-02-19T19:11:59.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.305+0000 7f7a6b7cb640 1 -- start start 2024-02-19T19:11:59.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.305+0000 7f7a6b7cb640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:59.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a6b7cb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64196210 0x7f7a641995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:59.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a6b7cb640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64196810 con 0x7f7a640675b0 2024-02-19T19:11:59.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a6b7cb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64196950 con 0x7f7a64196210 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56228/0 (socket says 172.21.15.39:56228) 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 -- 172.21.15.39:0/661215398 learned_addr learned my addr 172.21.15.39:0/661215398 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a68d66640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64196210 0x7f7a641995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 -- 172.21.15.39:0/661215398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64196210 msgr2=0x7f7a641995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64196210 0x7f7a641995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 -- 172.21.15.39:0/661215398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60009560 con 0x7f7a640675b0 2024-02-19T19:11:59.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a69567640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64195cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009230 tx=0x7f7a60004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:59.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.306+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a600044a0 con 0x7f7a640675b0 2024-02-19T19:11:59.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.307+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a64199b60 con 0x7f7a640675b0 2024-02-19T19:11:59.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.307+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7a600054b0 con 0x7f7a640675b0 2024-02-19T19:11:59.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.307+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a60034440 con 0x7f7a640675b0 2024-02-19T19:11:59.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.307+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6419a0c0 con 0x7f7a640675b0 2024-02-19T19:11:59.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.308+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6404e040 con 0x7f7a640675b0 2024-02-19T19:11:59.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.308+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 38) v1 ==== 91493+0+0 (secure 0 0 0) 0x7f7a60003780 con 0x7f7a640675b0 2024-02-19T19:11:59.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.309+0000 7f7a5a7fc640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7a5006d810 0x7f7a5006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:11:59.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.309+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7a600afa10 con 0x7f7a640675b0 2024-02-19T19:11:59.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.309+0000 7f7a68d66640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7a5006d810 0x7f7a5006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:11:59.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.310+0000 7f7a68d66640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7a5006d810 0x7f7a5006fcd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005d00 tx=0x7f7a54005c50 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:11:59.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.314+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a60079170 con 0x7f7a640675b0 2024-02-19T19:11:59.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:59 smithi039 ceph-mon[92224]: pgmap v211: 65 pgs: 65 active+clean; 161 MiB data, 771 MiB used, 536 GiB / 536 GiB avail; 687 KiB/s rd, 7.1 MiB/s wr, 666 op/s 2024-02-19T19:11:59.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:59 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/1181445820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:11:59.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:59 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:59.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:59 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/155817754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:11:59.499 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:11:59 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/4196015677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:11:59.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:59 smithi086 ceph-mon[82675]: pgmap v211: 65 pgs: 65 active+clean; 161 MiB data, 771 MiB used, 536 GiB / 536 GiB avail; 687 KiB/s rd, 7.1 MiB/s wr, 666 op/s 2024-02-19T19:11:59.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:59 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/1181445820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:11:59.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:59 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:11:59.536 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:59 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/155817754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:11:59.537 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:11:59 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/4196015677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:11:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.721+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7a64108590 con 0x7f7a640675b0 2024-02-19T19:11:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.722+0000 7f7a5a7fc640 1 -- 172.21.15.39:0/661215398 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 43 entries v85) v1 ==== 81+0+2541 (secure 0 0 0) 0x7f7a6007c830 con 0x7f7a640675b0 2024-02-19T19:11:59.745 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6826/978975893 2024-02-20T19:11:33.833848+0000 2024-02-19T19:11:59.745 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6825/2684130153 2024-02-20T19:11:07.184426+0000 2024-02-19T19:11:59.745 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6824/2684130153 2024-02-20T19:11:07.184426+0000 2024-02-19T19:11:59.745 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/3588485196 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.745 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6828/1232386891 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.745 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3933707157 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2610970659 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/728525375 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1444159856 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/4117360583 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/4293838126 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2615309126 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/1239352357 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2307886882 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:59.746 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6827/978975893 2024-02-20T19:11:33.833848+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2619661656 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/883565860 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6827/2541159330 2024-02-20T19:02:13.966534+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/728525375 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2039816448 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2158365942 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/913229036 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1749005598 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:59.747 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2039816448 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:6829/1232386891 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/2307886882 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1614648654 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/913229036 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2313120383 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/4112182782 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3613879910 2024-02-20T18:55:10.759262+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/1077741088 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2871115404 2024-02-20T18:55:43.704412+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2362951092 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3788984480 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3009513678 2024-02-20T19:04:44.155895+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/2158365942 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6826/2541159330 2024-02-20T19:02:13.966534+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1261064923 2024-02-20T18:57:00.645468+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1410355349 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.86:0/4267507691 2024-02-20T19:05:27.023863+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3608355132 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.749 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2027083659 2024-02-20T19:06:13.407336+0000 2024-02-19T19:11:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:listed 43 entries 2024-02-19T19:11:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7a5006d810 msgr2=0x7f7a5006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:59.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7a5006d810 0x7f7a5006fcd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005d00 tx=0x7f7a54005c50 comp rx=0 tx=0).stop 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 msgr2=0x7f7a64195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64195cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009230 tx=0x7f7a60004f00 comp rx=0 tx=0).stop 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 shutdown_connections 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:6800/517499402,v1:172.21.15.39:6801/517499402] conn(0x7f7a5006d810 0x7f7a5006fcd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a64196210 0x7f7a641995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 --2- 172.21.15.39:0/661215398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a640675b0 0x7f7a64195cd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:11:59.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 >> 172.21.15.39:0/661215398 conn(0x7f7a64075bb0 msgr2=0x7f7a640fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:11:59.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 shutdown_connections 2024-02-19T19:11:59.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:11:59.725+0000 7f7a6b7cb640 1 -- 172.21.15.39:0/661215398 wait complete. 2024-02-19T19:11:59.762 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-02-19T19:11:59.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:11:59.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-02-19T19:11:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:umount: /home/ubuntu/cephtest/mnt.1: target is busy. 2024-02-19T19:11:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-02-19T19:11:59.793 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.1... 2024-02-19T19:11:59.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:11:59.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2024-02-19T19:12:00.495 INFO:teuthology.orchestra.run.smithi086.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-02-19T19:12:00.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-02-19T19:12:00.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 597694 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.496 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.497 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.499 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 3w CHR 1,11 0t0 10 /dev/kmsg 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 4u a_inode 0,14 0 1076 [eventpoll:5,6,8,9,11,14,15,16,18,34,35,44,45,48,51,52,53,54,60,72,73,77,78,79,81,82,85,87,88,89,90,114...] 2024-02-19T19:12:00.500 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 5u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 6r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 7r DIR 0,25 0 1 /sys/fs/cgroup 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 8u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 9u a_inode 0,14 0 1076 [eventpoll:10,13] 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 10r REG 0,20 0 1171 /proc/1/mountinfo 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 11r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 12u netlink 0t0 16611 AUDIT 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 13r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 14r REG 0,20 0 4026532076 /proc/swaps 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 15u netlink 0t0 16603 KOBJECT_UEVENT 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 16u unix 0xffffa09a884a9400 0t0 16605 /run/systemd/private type=STREAM (LISTEN) 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 17r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 19r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 20u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 21u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.502 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 22u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 23u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 24u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 25u unix 0xffffa09a88ebad00 0t0 20642 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 26u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 27r CHR 10,235 0t0 111 /dev/autofs 2024-02-19T19:12:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 28u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 29u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 30u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 31u a_inode 0,14 0 1076 bpf-prog 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 32u unix 0xffffa09a81d54100 0t0 253592 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 34u unix 0xffffa09a887c5500 0t0 1173 /run/systemd/notify type=DGRAM (CONNECTED) 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 35u unix 0xffffa09a887c4600 0t0 1174 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 36u unix 0xffffa09a887c3200 0t0 1175 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.504 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 39u unix 0xffffa09a81d57800 0t0 253580 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.505 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 44u unix 0xffffa09a84b25500 0t0 1179 /run/systemd/io.system.ManagedOOM type=STREAM (LISTEN) 2024-02-19T19:12:00.505 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 45u unix 0xffffa09a84b27300 0t0 1178 /run/systemd/userdb/io.systemd.DynamicUser type=STREAM (LISTEN) 2024-02-19T19:12:00.540 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 48u unix 0xffffa09a88650000 0t0 20586 /run/lvm/lvmpolld.socket type=STREAM (LISTEN) 2024-02-19T19:12:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 49u unix 0xffffa09a887c5000 0t0 1188 /run/systemd/journal/socket type=DGRAM (CONNECTED) 2024-02-19T19:12:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 50u unix 0xffffa09a84b23c00 0t0 1190 /run/systemd/journal/stdout type=STREAM (LISTEN) 2024-02-19T19:12:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 51u unix 0xffffa09a88eba300 0t0 20598 /run/systemd/coredump type=SEQPACKET (LISTEN) 2024-02-19T19:12:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 52u FIFO 0,24 0t0 453 /run/initctl 2024-02-19T19:12:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 53u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.541 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 54u CHR 10,242 0t0 165 /dev/rfkill 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 55u netlink 0t0 20602 KOBJECT_UEVENT 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 57u unix 0xffffa09a88ebee00 0t0 20600 /run/udev/control type=SEQPACKET (LISTEN) 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 58u unix 0xffffa09a887c0000 0t0 1186 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 60u unix 0xffffa09a81d57300 0t0 23049 type=STREAM (CONNECTED) 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 72u FIFO 0,24 0t0 447 /run/dmeventd-server 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 73u FIFO 0,24 0t0 448 /run/dmeventd-client 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 74u unix 0xffffa09a81d56400 0t0 23041 /run/dbus/system_bus_socket type=STREAM (LISTEN) 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 76u unix 0xffffa09a88654b00 0t0 102365 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.542 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 77u unix 0xffffa09a88654600 0t0 19677 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 78u unix 0xffffa09a88653200 0t0 19893 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 79u unix 0xffffa09a8874d000 0t0 19928 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 80u unix 0xffffa09a884a9900 0t0 19935 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 81u unix 0xffffa09a884ac600 0t0 19938 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 82u unix 0xffffa09a887e0500 0t0 19940 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 83u unix 0xffffa09a887e2800 0t0 19941 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 84u unix 0xffffa09a884af800 0t0 19945 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 85u unix 0xffffa09a887e6e00 0t0 20028 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.543 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 86u unix 0xffffa09a88653c00 0t0 21726 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 87u unix 0xffffa09a81d52800 0t0 658 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 88u unix 0xffffa09a8874c100 0t0 20037 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 89u unix 0xffffa09a884aad00 0t0 23360 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 90u unix 0xffffa09a84b26900 0t0 92780 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 114r FIFO 0,13 0t0 1334 pipe 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 115u unix 0xffffa09a88653700 0t0 20588 /run/rpcbind.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 116u IPv4 19623 0t0 TCP *:sunrpc (LISTEN) 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 117u IPv4 1341 0t0 UDP *:sunrpc 2024-02-19T19:12:00.544 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 118u IPv6 19629 0t0 TCP *:sunrpc (LISTEN) 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 119u IPv6 19635 0t0 UDP *:sunrpc 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1 root 120u unix 0xffffa09a81d55500 0t0 23043 /run/.heim_org.h5l.kcm-socket type=STREAM (LISTEN) 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:kthreadd 2 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:kthreadd 2 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:kthreadd 2 root txt unknown /proc/2/exe 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_gp 3 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_gp 3 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_gp 3 root txt unknown /proc/3/exe 2024-02-19T19:12:00.545 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_par_g 4 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_par_g 4 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_par_g 4 root txt unknown /proc/4/exe 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:slub_flus 5 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:slub_flus 5 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:slub_flus 5 root txt unknown /proc/5/exe 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:netns 6 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:netns 6 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.546 INFO:teuthology.orchestra.run.smithi086.stdout:netns 6 root txt unknown /proc/6/exe 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 8 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 8 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 8 root txt unknown /proc/8/exe 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 9 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 9 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 9 root txt unknown /proc/9/exe 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:mm_percpu 10 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:mm_percpu 10 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.547 INFO:teuthology.orchestra.run.smithi086.stdout:mm_percpu 10 root txt unknown /proc/10/exe 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 11 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 11 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 11 root txt unknown /proc/11/exe 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 12 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 12 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 12 root txt unknown /proc/12/exe 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 13 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 13 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 13 root txt unknown /proc/13/exe 2024-02-19T19:12:00.548 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 14 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 14 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_tasks 14 root txt unknown /proc/14/exe 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 15 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 15 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 15 root txt unknown /proc/15/exe 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:pr/ttyS1 16 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:pr/ttyS1 16 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.549 INFO:teuthology.orchestra.run.smithi086.stdout:pr/ttyS1 16 root txt unknown /proc/16/exe 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_preem 17 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_preem 17 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:rcu_preem 17 root txt unknown /proc/17/exe 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:migration 18 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:migration 18 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:migration 18 root txt unknown /proc/18/exe 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 19 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.550 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 19 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 19 root txt unknown /proc/19/exe 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/0 21 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/0 21 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/0 21 root txt unknown /proc/21/exe 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/1 22 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/1 22 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/1 22 root txt unknown /proc/22/exe 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 23 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 23 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.551 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 23 root txt unknown /proc/23/exe 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:migration 24 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:migration 24 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:migration 24 root txt unknown /proc/24/exe 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 25 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 25 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 25 root txt unknown /proc/25/exe 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 27 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 27 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.552 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 27 root txt unknown /proc/27/exe 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/2 28 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/2 28 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/2 28 root txt unknown /proc/28/exe 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 29 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 29 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 29 root txt unknown /proc/29/exe 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:migration 30 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:migration 30 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:migration 30 root txt unknown /proc/30/exe 2024-02-19T19:12:00.553 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 31 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 31 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 31 root txt unknown /proc/31/exe 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 33 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 33 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 33 root txt unknown /proc/33/exe 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/3 34 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/3 34 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/3 34 root txt unknown /proc/34/exe 2024-02-19T19:12:00.554 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 35 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 35 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 35 root txt unknown /proc/35/exe 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:migration 36 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:migration 36 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:migration 36 root txt unknown /proc/36/exe 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 37 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 37 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 37 root txt unknown /proc/37/exe 2024-02-19T19:12:00.555 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 39 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 39 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 39 root txt unknown /proc/39/exe 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/4 40 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/4 40 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/4 40 root txt unknown /proc/40/exe 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 41 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 41 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.556 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 41 root txt unknown /proc/41/exe 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:migration 42 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:migration 42 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:migration 42 root txt unknown /proc/42/exe 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 43 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 43 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 43 root txt unknown /proc/43/exe 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 45 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 45 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 45 root txt unknown /proc/45/exe 2024-02-19T19:12:00.557 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/5 46 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/5 46 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/5 46 root txt unknown /proc/46/exe 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 47 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 47 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 47 root txt unknown /proc/47/exe 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:migration 48 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:migration 48 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.558 INFO:teuthology.orchestra.run.smithi086.stdout:migration 48 root txt unknown /proc/48/exe 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 49 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 49 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 49 root txt unknown /proc/49/exe 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 51 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 51 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 51 root txt unknown /proc/51/exe 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/6 52 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/6 52 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.559 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/6 52 root txt unknown /proc/52/exe 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 53 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 53 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 53 root txt unknown /proc/53/exe 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:migration 54 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:migration 54 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:migration 54 root txt unknown /proc/54/exe 2024-02-19T19:12:00.560 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 55 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 55 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 55 root txt unknown /proc/55/exe 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 57 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 57 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 57 root txt unknown /proc/57/exe 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/7 58 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/7 58 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:cpuhp/7 58 root txt unknown /proc/58/exe 2024-02-19T19:12:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 59 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 59 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:idle_inje 59 root txt unknown /proc/59/exe 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:migration 60 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:migration 60 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:migration 60 root txt unknown /proc/60/exe 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 61 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 61 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:ksoftirqd 61 root txt unknown /proc/61/exe 2024-02-19T19:12:00.562 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 63 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 63 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 63 root txt unknown /proc/63/exe 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 65 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 65 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 65 root txt unknown /proc/65/exe 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 66 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 66 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 66 root txt unknown /proc/66/exe 2024-02-19T19:12:00.563 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 69 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 69 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 69 root txt unknown /proc/69/exe 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 70 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 70 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 70 root txt unknown /proc/70/exe 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kdevtmpfs 71 root cwd DIR 0,5 3500 1 / 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kdevtmpfs 71 root rtd DIR 0,5 3500 1 / 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:kdevtmpfs 71 root txt unknown /proc/71/exe 2024-02-19T19:12:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:inet_frag 72 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:inet_frag 72 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:inet_frag 72 root txt unknown /proc/72/exe 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:kauditd 73 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:kauditd 73 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:kauditd 73 root txt unknown /proc/73/exe 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:khungtask 75 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:khungtask 75 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:khungtask 75 root txt unknown /proc/75/exe 2024-02-19T19:12:00.565 INFO:teuthology.orchestra.run.smithi086.stdout:oom_reape 76 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:oom_reape 76 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:oom_reape 76 root txt unknown /proc/76/exe 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:writeback 77 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:writeback 77 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:writeback 77 root txt unknown /proc/77/exe 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:kcompactd 78 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:kcompactd 78 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:kcompactd 78 root txt unknown /proc/78/exe 2024-02-19T19:12:00.566 INFO:teuthology.orchestra.run.smithi086.stdout:ksmd 79 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:ksmd 79 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:ksmd 79 root txt unknown /proc/79/exe 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:khugepage 81 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:khugepage 81 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:khugepage 81 root txt unknown /proc/81/exe 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:cryptd 82 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:cryptd 82 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:cryptd 82 root txt unknown /proc/82/exe 2024-02-19T19:12:00.567 INFO:teuthology.orchestra.run.smithi086.stdout:kintegrit 83 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:kintegrit 83 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:kintegrit 83 root txt unknown /proc/83/exe 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:kblockd 84 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:kblockd 84 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:kblockd 84 root txt unknown /proc/84/exe 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:blkcg_pun 85 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:blkcg_pun 85 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.568 INFO:teuthology.orchestra.run.smithi086.stdout:blkcg_pun 85 root txt unknown /proc/85/exe 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:tpm_dev_w 87 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:tpm_dev_w 87 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:tpm_dev_w 87 root txt unknown /proc/87/exe 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:md 88 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:md 88 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:md 88 root txt unknown /proc/88/exe 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:md_bitmap 89 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.569 INFO:teuthology.orchestra.run.smithi086.stdout:md_bitmap 89 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:md_bitmap 89 root txt unknown /proc/89/exe 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:edac-poll 90 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:edac-poll 90 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:edac-poll 90 root txt unknown /proc/90/exe 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:watchdogd 91 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:watchdogd 91 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:watchdogd 91 root txt unknown /proc/91/exe 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 93 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.570 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 93 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 93 root txt unknown /proc/93/exe 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kswapd0 95 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kswapd0 95 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kswapd0 95 root txt unknown /proc/95/exe 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kthrotld 100 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kthrotld 100 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:kthrotld 100 root txt unknown /proc/100/exe 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:acpi_ther 109 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.571 INFO:teuthology.orchestra.run.smithi086.stdout:acpi_ther 109 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:acpi_ther 109 root txt unknown /proc/109/exe 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:kmpath_rd 110 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:kmpath_rd 110 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:kmpath_rd 110 root txt unknown /proc/110/exe 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:kaluad 111 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:kaluad 111 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:kaluad 111 root txt unknown /proc/111/exe 2024-02-19T19:12:00.572 INFO:teuthology.orchestra.run.smithi086.stdout:mld 112 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:mld 112 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:mld 112 root txt unknown /proc/112/exe 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 113 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 113 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 113 root txt unknown /proc/113/exe 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:ipv6_addr 114 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:ipv6_addr 114 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:ipv6_addr 114 root txt unknown /proc/114/exe 2024-02-19T19:12:00.573 INFO:teuthology.orchestra.run.smithi086.stdout:kstrp 123 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:kstrp 123 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:kstrp 123 root txt unknown /proc/123/exe 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:zswap-shr 130 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:zswap-shr 130 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:zswap-shr 130 root txt unknown /proc/130/exe 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 131 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 131 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 131 root txt unknown /proc/131/exe 2024-02-19T19:12:00.574 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 238 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 238 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 238 root txt unknown /proc/238/exe 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 278 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 278 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 278 root txt unknown /proc/278/exe 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 281 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 281 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.575 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 281 root txt unknown /proc/281/exe 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 283 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 283 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 283 root txt unknown /proc/283/exe 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 319 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 319 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 319 root txt unknown /proc/319/exe 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 355 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 355 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 355 root txt unknown /proc/355/exe 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:ata_sff 430 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:ata_sff 430 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:ata_sff 430 root txt unknown /proc/430/exe 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:ixgbe 432 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:ixgbe 432 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:ixgbe 432 root txt unknown /proc/432/exe 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-wq 435 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.577 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-wq 435 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-wq 435 root txt unknown /proc/435/exe 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-rese 436 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-rese 436 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-rese 436 root txt unknown /proc/436/exe 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-dele 437 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-dele 437 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-dele 437 root txt unknown /proc/437/exe 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-auth 438 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.578 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-auth 438 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:nvme-auth 438 root txt unknown /proc/438/exe 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_0 442 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_0 442 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_0 442 root txt unknown /proc/442/exe 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 443 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 443 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 443 root txt unknown /proc/443/exe 2024-02-19T19:12:00.579 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_1 444 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_1 444 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_1 444 root txt unknown /proc/444/exe 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 445 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 445 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 445 root txt unknown /proc/445/exe 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_2 446 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_2 446 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.580 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_2 446 root txt unknown /proc/446/exe 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 447 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 447 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 447 root txt unknown /proc/447/exe 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_3 448 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_3 448 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_3 448 root txt unknown /proc/448/exe 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 449 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 449 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.581 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 449 root txt unknown /proc/449/exe 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_4 454 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_4 454 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_4 454 root txt unknown /proc/454/exe 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 455 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 455 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 455 root txt unknown /proc/455/exe 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_5 456 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_5 456 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_5 456 root txt unknown /proc/456/exe 2024-02-19T19:12:00.582 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 457 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 457 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 457 root txt unknown /proc/457/exe 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_6 458 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_6 458 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_6 458 root txt unknown /proc/458/exe 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 459 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 459 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 459 root txt unknown /proc/459/exe 2024-02-19T19:12:00.583 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_7 460 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_7 460 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_7 460 root txt unknown /proc/460/exe 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 461 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 461 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 461 root txt unknown /proc/461/exe 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_8 462 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_8 462 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.584 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_8 462 root txt unknown /proc/462/exe 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 463 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 463 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 463 root txt unknown /proc/463/exe 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_9 464 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_9 464 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_eh_9 464 root txt unknown /proc/464/exe 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 465 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.585 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 465 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:scsi_tmf_ 465 root txt unknown /proc/465/exe 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:jbd2/sda1 499 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:jbd2/sda1 499 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:jbd2/sda1 499 root txt unknown /proc/499/exe 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:ext4-rsv- 500 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:ext4-rsv- 500 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:ext4-rsv- 500 root txt unknown /proc/500/exe 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root txt REG 8,1 185496 20360 /usr/lib/systemd/systemd-journald 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.587 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.588 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.590 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root mem REG 0,24 8 55 /run/systemd/journal/kernel-seqnum 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 3u unix 0xffffa09a887c0000 0t0 1186 /run/systemd/journal/dev-log type=DGRAM (CONNECTED) 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 4u unix 0xffffa09a887c5000 0t0 1188 /run/systemd/journal/socket type=DGRAM (CONNECTED) 2024-02-19T19:12:00.591 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 5u unix 0xffffa09a84b23c00 0t0 1190 /run/systemd/journal/stdout type=STREAM (LISTEN) 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 6w CHR 1,11 0t0 10 /dev/kmsg 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 7u a_inode 0,14 0 1076 [eventpoll:3,4,5,8,9,10,11,12,13,15,17,18,19,20,22,23,24,25,26,27,28,29,30,31,32,33,34] 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 8u CHR 1,11 0t0 10 /dev/kmsg 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 9u unix 0xffffa09a884a8f00 0t0 16635 /run/systemd/journal/io.systemd.journal type=STREAM (LISTEN) 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 10r REG 0,20 0 350 /proc/sys/kernel/hostname 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 11u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 12u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 13u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.592 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 14u unix 0xffffa09cb10b8500 0t0 16636 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 15u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 16u REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 17u unix 0xffffa09a88654600 0t0 19677 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 18u unix 0xffffa09a8874d000 0t0 19928 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 19u unix 0xffffa09a884a9900 0t0 19935 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 20u unix 0xffffa09a88653200 0t0 19893 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 21u REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 22u unix 0xffffa09a88654b00 0t0 102365 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.593 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 23u unix 0xffffa09a884ac600 0t0 19938 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 24u unix 0xffffa09a887e0500 0t0 19940 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 25u unix 0xffffa09a887e2800 0t0 19941 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 26u unix 0xffffa09a84b26900 0t0 92780 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 27u unix 0xffffa09a887e6e00 0t0 20028 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 28u unix 0xffffa09a884af800 0t0 19945 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 29u unix 0xffffa09a88653c00 0t0 21726 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 30u unix 0xffffa09a81d52800 0t0 658 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 31u unix 0xffffa09a81d57800 0t0 253580 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.594 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 32u unix 0xffffa09a8874c100 0t0 20037 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 33u unix 0xffffa09a884aad00 0t0 23360 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-j 577 root 34u unix 0xffffa09a81d54100 0t0 253592 /run/systemd/journal/stdout type=STREAM (CONNECTED) 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:ext4lazyi 583 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:ext4lazyi 583 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:ext4lazyi 583 root txt unknown /proc/583/exe 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root txt REG 8,1 597744 11595 /usr/bin/udevadm 2024-02-19T19:12:00.595 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 11730661 64128 /etc/udev/hwdb.bin 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 597694 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 585432 265291 /usr/lib/modules/5.14.0-421.el9.x86_64/modules.symbols.bin 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 829766 265288 /usr/lib/modules/5.14.0-421.el9.x86_64/modules.alias.bin 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 397119 265297 /usr/lib/modules/5.14.0-421.el9.x86_64/modules.dep.bin 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.596 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.598 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 11541 265292 /usr/lib/modules/5.14.0-421.el9.x86_64/modules.builtin.bin 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.599 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 1u unix 0xffffa09a88650f00 0t0 20690 type=STREAM (CONNECTED) 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 2u unix 0xffffa09a88650f00 0t0 20690 type=STREAM (CONNECTED) 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 3u unix 0xffffa09a88ebee00 0t0 20600 /run/udev/control type=SEQPACKET (LISTEN) 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 4u netlink 0t0 20602 KOBJECT_UEVENT 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 5u unix 0xffffa09a887c1e00 0t0 1369 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 6u unix 0xffffa09a887c4100 0t0 1373 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.600 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 7u unix 0xffffa09a887c4b00 0t0 1374 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 9u a_inode 0,14 0 1076 [eventpoll:3,4,6,8,10,11,14] 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 10u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 11u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 12r REG 8,1 11730661 64128 /etc/udev/hwdb.bin 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-u 597 root 14u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:ipmi-msgh 639 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.601 INFO:teuthology.orchestra.run.smithi086.stdout:ipmi-msgh 639 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:ipmi-msgh 639 root txt unknown /proc/639/exe 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:kipmi0 647 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:kipmi0 647 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:kipmi0 647 root txt unknown /proc/647/exe 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.602 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc txt REG 8,1 61200 14999 /usr/bin/rpcbind 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.603 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.604 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 1u unix 0xffffa09a88654100 0t0 20766 type=STREAM (CONNECTED) 2024-02-19T19:12:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 2u unix 0xffffa09a88654100 0t0 20766 type=STREAM (CONNECTED) 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 3u unix 0xffffa09a88653700 0t0 20588 /run/rpcbind.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 4u IPv4 19623 0t0 TCP *:sunrpc (LISTEN) 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 5u IPv4 1341 0t0 UDP *:sunrpc 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 6u IPv6 19629 0t0 TCP *:sunrpc (LISTEN) 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 7u IPv6 19635 0t0 UDP *:sunrpc 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpcbind 673 rpc 8rW REG 0,24 0 1083 /run/rpcbind/rpcbind.lock 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpciod 676 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.606 INFO:teuthology.orchestra.run.smithi086.stdout:rpciod 676 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:rpciod 676 root txt unknown /proc/676/exe 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:xprtiod 677 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:xprtiod 677 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:xprtiod 677 root txt unknown /proc/677/exe 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root txt REG 8,1 144576 11776 /usr/sbin/auditd 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.607 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.608 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.609 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 3u netlink 0t0 542 AUDIT 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 4w REG 8,1 3656407 64146 /var/log/audit/audit.log 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 6u unix 0xffffa09a883d5a00 0t0 545 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 7u unix 0xffffa09a86e48f00 0t0 544 type=STREAM (CONNECTED) 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 8u unix 0xffffa09a86e4ee00 0t0 546 type=STREAM (CONNECTED) 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 9u unix 0xffffa09a86e4c100 0t0 547 type=STREAM (CONNECTED) 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 root 10u a_inode 0,14 0 1076 [eventfd:0] 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.610 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root txt REG 8,1 144576 11776 /usr/sbin/auditd 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.611 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.612 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 3u netlink 0t0 542 AUDIT 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 4w REG 8,1 3656407 64146 /var/log/audit/audit.log 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 6u unix 0xffffa09a883d5a00 0t0 545 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 7u unix 0xffffa09a86e48f00 0t0 544 type=STREAM (CONNECTED) 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 8u unix 0xffffa09a86e4ee00 0t0 546 type=STREAM (CONNECTED) 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 9u unix 0xffffa09a86e4c100 0t0 547 type=STREAM (CONNECTED) 2024-02-19T19:12:00.613 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 679 auditd root 10u a_inode 0,14 0 1076 [eventfd:0] 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root txt REG 8,1 144576 11776 /usr/sbin/auditd 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.614 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.615 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 3u netlink 0t0 542 AUDIT 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 4w REG 8,1 3656407 64146 /var/log/audit/audit.log 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 6u unix 0xffffa09a883d5a00 0t0 545 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 7u unix 0xffffa09a86e48f00 0t0 544 type=STREAM (CONNECTED) 2024-02-19T19:12:00.616 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 8u unix 0xffffa09a86e4ee00 0t0 546 type=STREAM (CONNECTED) 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 9u unix 0xffffa09a86e4c100 0t0 547 type=STREAM (CONNECTED) 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:auditd 678 681 auditd root 10u a_inode 0,14 0 1076 [eventfd:0] 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root txt REG 8,1 19624 11914 /usr/sbin/sedispatch 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.617 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.618 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 143568 9730 /usr/lib64/libauparse.so.0.0.0 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root 0u unix 0xffffa09a86e4c600 0t0 543 type=STREAM (CONNECTED) 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:sedispatc 680 root 3u unix 0xffffa09a84b20a00 0t0 1936 type=STREAM (CONNECTED) 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.619 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus txt REG 8,1 131896 11213 /usr/bin/dbus-broker-launch 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus DEL REG 0,1 2048 /memfd:dbus-broker-log 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.620 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.621 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 1u unix 0xffffa09a88748a00 0t0 18925 type=STREAM (CONNECTED) 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 2u unix 0xffffa09a88748a00 0t0 18925 type=STREAM (CONNECTED) 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 3u unix 0xffffa09a81d56400 0t0 23041 /run/dbus/system_bus_socket type=STREAM (LISTEN) 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 4u unix 0xffffa09a8875cb00 0t0 18935 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 5u a_inode 0,14 0 1076 [eventpoll:6,7,8,9,10,11] 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 6u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.622 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 7r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 8u unix 0xffffa09a8874f800 0t0 18937 type=STREAM (CONNECTED) 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 9u unix 0xffffa09a8874a300 0t0 18939 type=STREAM (CONNECTED) 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 10u a_inode 0,14 0 1076 [pidfd:718] 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 11u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 717 dbus 12u REG 0,1 2097152 2048 /memfd:dbus-broker-log (deleted) 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.623 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus txt REG 8,1 236336 11212 /usr/bin/dbus-broker 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.624 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 1u unix 0xffffa09a88748a00 0t0 18925 type=STREAM (CONNECTED) 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 2u unix 0xffffa09a88748a00 0t0 18925 type=STREAM (CONNECTED) 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 3u sock 0,8 0t0 572 protocol: NETLINK 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 4u unix 0xffffa09a8875cb00 0t0 18935 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 5u a_inode 0,14 0 1076 [eventpoll:6,7,8,9,10,11,12,13,14,15,17,18,19,20,21] 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 6u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.625 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 7u unix 0xffffa09a81d56400 0t0 23041 /run/dbus/system_bus_socket type=STREAM (LISTEN) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 8u unix 0xffffa09a81d53200 0t0 573 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 9u unix 0xffffa09a88749e00 0t0 18938 type=STREAM (CONNECTED) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 10u unix 0xffffa09a8874e900 0t0 575 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 11u unix 0xffffa09a884aa300 0t0 580 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 12u unix 0xffffa09a81d51900 0t0 18981 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 13u unix 0xffffa09a86e4e400 0t0 18983 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.626 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 14u unix 0xffffa09a81d56e00 0t0 255640 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 15u unix 0xffffa09a8874e400 0t0 21703 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 17u unix 0xffffa09a84b25000 0t0 19239 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 18u unix 0xffffa09a88748f00 0t0 1538 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 19u unix 0xffffa09a884ae400 0t0 21847 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 20u unix 0xffffa09a88651400 0t0 104335 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:dbus-brok 718 dbus 21u unix 0xffffa09a81d52300 0t0 255674 /run/dbus/system_bus_socket type=STREAM (CONNECTED) 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.627 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root txt REG 8,1 3644720 11633 /usr/sbin/NetworkManager 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 317552 52828 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-settings-plugin-ifcfg-rh.so 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 62456 9782 /usr/lib64/libjansson.so.4.14.0 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 31792 10954 /usr/lib64/libteamdctl.so.0.1.5 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 57016 52811 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-device-plugin-team.so 2024-02-19T19:12:00.628 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 138712 9890 /usr/lib64/libbrotlicommon.so.1.0.9 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.629 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 53048 9892 /usr/lib64/libbrotlidec.so.1.0.9 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.630 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 77488 9888 /usr/lib64/libpsl.so.5.3.3 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 483672 11481 /usr/lib64/libssh.so.4.9.4 2024-02-19T19:12:00.631 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 169704 11377 /usr/lib64/libnghttp2.so.14.20.1 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.632 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.633 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 662304 11485 /usr/lib64/libcurl.so.4.7.0 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 32408 11334 /usr/lib64/libndp.so.0.2.0 2024-02-19T19:12:00.634 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.635 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 1u unix 0xffffa09a884aee00 0t0 16744 type=STREAM (CONNECTED) 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 2u unix 0xffffa09a884aee00 0t0 16744 type=STREAM (CONNECTED) 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 3u a_inode 0,14 0 1076 [eventfd:5] 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 4u a_inode 0,14 0 1076 [eventfd:6] 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 5u unix 0xffffa09a818af300 0t0 23092 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 6u unix 0xffffa09a81d50500 0t0 23093 type=STREAM (CONNECTED) 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 7u a_inode 0,14 0 1076 [eventfd:7] 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 8r REG 0,4 0 4026531840 net 2024-02-19T19:12:00.636 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 9r REG 0,4 0 4026532495 mnt 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 10u netlink 0t0 23097 KOBJECT_UEVENT 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 11u netlink 0t0 23098 GENERIC 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 12u netlink 0t0 23099 GENERIC 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 13u netlink 0t0 23100 ROUTE 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 14r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 15r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 16u netlink 0t0 23133 KOBJECT_UEVENT 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 17u unix 0xffffa09a8874bc00 0t0 18987 type=STREAM (CONNECTED) 2024-02-19T19:12:00.637 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 18u a_inode 0,14 0 1076 [eventpoll] 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 19u a_inode 0,14 0 1076 [eventpoll:21,26] 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 20w FIFO 0,24 0t0 1168 /run/systemd/inhibit/1.ref 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 23u a_inode 0,14 0 1076 [eventpoll:24,25] 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 24u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 25u pack 21756 0t0 ARP type=SOCK_DGRAM 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 root 26u IPv4 21757 0t0 UDP smithi086.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2024-02-19T19:12:00.638 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root txt REG 8,1 3644720 11633 /usr/sbin/NetworkManager 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 317552 52828 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-settings-plugin-ifcfg-rh.so 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 62456 9782 /usr/lib64/libjansson.so.4.14.0 2024-02-19T19:12:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 31792 10954 /usr/lib64/libteamdctl.so.0.1.5 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 57016 52811 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-device-plugin-team.so 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 138712 9890 /usr/lib64/libbrotlicommon.so.1.0.9 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:00.640 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.641 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 53048 9892 /usr/lib64/libbrotlidec.so.1.0.9 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.642 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 77488 9888 /usr/lib64/libpsl.so.5.3.3 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 483672 11481 /usr/lib64/libssh.so.4.9.4 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 169704 11377 /usr/lib64/libnghttp2.so.14.20.1 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.643 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.644 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 662304 11485 /usr/lib64/libcurl.so.4.7.0 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.645 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 32408 11334 /usr/lib64/libndp.so.0.2.0 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.646 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 1u unix 0xffffa09a884aee00 0t0 16744 type=STREAM (CONNECTED) 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 2u unix 0xffffa09a884aee00 0t0 16744 type=STREAM (CONNECTED) 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 3u a_inode 0,14 0 1076 [eventfd:5] 2024-02-19T19:12:00.647 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 4u a_inode 0,14 0 1076 [eventfd:6] 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 5u unix 0xffffa09a818af300 0t0 23092 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 6u unix 0xffffa09a81d50500 0t0 23093 type=STREAM (CONNECTED) 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 7u a_inode 0,14 0 1076 [eventfd:7] 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 8r REG 0,4 0 4026531840 net 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 9r REG 0,4 0 4026532495 mnt 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 10u netlink 0t0 23097 KOBJECT_UEVENT 2024-02-19T19:12:00.648 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 11u netlink 0t0 23098 GENERIC 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 12u netlink 0t0 23099 GENERIC 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 13u netlink 0t0 23100 ROUTE 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 14r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 15r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 16u netlink 0t0 23133 KOBJECT_UEVENT 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 17u unix 0xffffa09a8874bc00 0t0 18987 type=STREAM (CONNECTED) 2024-02-19T19:12:00.649 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 18u a_inode 0,14 0 1076 [eventpoll] 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 19u a_inode 0,14 0 1076 [eventpoll:21,26] 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 20w FIFO 0,24 0t0 1168 /run/systemd/inhibit/1.ref 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 23u a_inode 0,14 0 1076 [eventpoll:24,25] 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 24u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 25u pack 21756 0t0 ARP type=SOCK_DGRAM 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 738 gmain root 26u IPv4 21757 0t0 UDP smithi086.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.650 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root txt REG 8,1 3644720 11633 /usr/sbin/NetworkManager 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 317552 52828 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-settings-plugin-ifcfg-rh.so 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 62456 9782 /usr/lib64/libjansson.so.4.14.0 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 31792 10954 /usr/lib64/libteamdctl.so.0.1.5 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 57016 52811 /usr/lib64/NetworkManager/1.44.0-2.el9/libnm-device-plugin-team.so 2024-02-19T19:12:00.651 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 138712 9890 /usr/lib64/libbrotlicommon.so.1.0.9 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.652 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 53048 9892 /usr/lib64/libbrotlidec.so.1.0.9 2024-02-19T19:12:00.653 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 77488 9888 /usr/lib64/libpsl.so.5.3.3 2024-02-19T19:12:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 483672 11481 /usr/lib64/libssh.so.4.9.4 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 169704 11377 /usr/lib64/libnghttp2.so.14.20.1 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.656 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 662304 11485 /usr/lib64/libcurl.so.4.7.0 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-02-19T19:12:00.657 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 32408 11334 /usr/lib64/libndp.so.0.2.0 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.658 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 1u unix 0xffffa09a884aee00 0t0 16744 type=STREAM (CONNECTED) 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 2u unix 0xffffa09a884aee00 0t0 16744 type=STREAM (CONNECTED) 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 3u a_inode 0,14 0 1076 [eventfd:5] 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 4u a_inode 0,14 0 1076 [eventfd:6] 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 5u unix 0xffffa09a818af300 0t0 23092 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.659 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 6u unix 0xffffa09a81d50500 0t0 23093 type=STREAM (CONNECTED) 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 7u a_inode 0,14 0 1076 [eventfd:7] 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 8r REG 0,4 0 4026531840 net 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 9r REG 0,4 0 4026532495 mnt 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 10u netlink 0t0 23097 KOBJECT_UEVENT 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 11u netlink 0t0 23098 GENERIC 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 12u netlink 0t0 23099 GENERIC 2024-02-19T19:12:00.660 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 13u netlink 0t0 23100 ROUTE 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 14r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 15r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 16u netlink 0t0 23133 KOBJECT_UEVENT 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 17u unix 0xffffa09a8874bc00 0t0 18987 type=STREAM (CONNECTED) 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 18u a_inode 0,14 0 1076 [eventpoll] 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 19u a_inode 0,14 0 1076 [eventpoll:21,26] 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 20w FIFO 0,24 0t0 1168 /run/systemd/inhibit/1.ref 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.661 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 23u a_inode 0,14 0 1076 [eventpoll:24,25] 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 24u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 25u pack 21756 0t0 ARP type=SOCK_DGRAM 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:NetworkMa 719 739 gdbus root 26u IPv4 21757 0t0 UDP smithi086.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root txt REG 8,1 61904 10507 /usr/sbin/irqbalance 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.662 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root DEL REG 8,1 9998 /usr/lib64/libnuma.so.1.0.0 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 1u unix 0xffffa09a884a8500 0t0 16753 type=STREAM (CONNECTED) 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 2u unix 0xffffa09a884a8500 0t0 16753 type=STREAM (CONNECTED) 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 3u a_inode 0,14 0 1076 [eventfd:1] 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 4u a_inode 0,14 0 1076 [eventfd:2] 2024-02-19T19:12:00.663 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 5u unix 0xffffa09cb1843c00 0t0 19946 /run/irqbalance/irqbalance722.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 root 6u unix 0xffffa09a83ada300 0t0 20217 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root txt REG 8,1 61904 10507 /usr/sbin/irqbalance 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.664 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root DEL REG 8,1 9998 /usr/lib64/libnuma.so.1.0.0 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 1u unix 0xffffa09a884a8500 0t0 16753 type=STREAM (CONNECTED) 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 2u unix 0xffffa09a884a8500 0t0 16753 type=STREAM (CONNECTED) 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 3u a_inode 0,14 0 1076 [eventfd:1] 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 4u a_inode 0,14 0 1076 [eventfd:2] 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 5u unix 0xffffa09cb1843c00 0t0 19946 /run/irqbalance/irqbalance722.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.665 INFO:teuthology.orchestra.run.smithi086.stdout:irqbalanc 722 729 gmain root 6u unix 0xffffa09a83ada300 0t0 20217 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt txt REG 8,1 32280 11292 /usr/bin/lsmd 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt mem REG 8,1 56912 10811 /usr/lib64/libconfig.so.11.0.2 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.666 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt 1u unix 0xffffa09a887e1400 0t0 21687 type=STREAM (CONNECTED) 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt 2u unix 0xffffa09a887e1400 0t0 21687 type=STREAM (CONNECTED) 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt 3uW REG 0,24 0 1121 /run/lsm/ipc/.lsmd-ipc-lock 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt 5u unix 0xffffa09a86e4b200 0t0 579 /var/run/lsm/ipc/simc type=STREAM (LISTEN) 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:lsmd 723 libstoragemgmt 6u unix 0xffffa09a86e4f300 0t0 595 /var/run/lsm/ipc/sim type=STREAM (LISTEN) 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root txt REG 8,1 194568 12461 /usr/sbin/mcelog 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.667 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root 1u unix 0xffffa09a887e1900 0t0 21690 type=STREAM (CONNECTED) 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root 2u unix 0xffffa09a887e1900 0t0 21690 type=STREAM (CONNECTED) 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root 3r CHR 10,227 0t0 142 /dev/mcelog 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:mcelog 724 root 4u unix 0xffffa09a887e4600 0t0 21696 /var/run/mcelog-client type=STREAM (LISTEN) 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.668 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.669 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.670 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.671 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.671 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.671 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.671 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.671 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.672 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.673 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.674 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.675 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.678 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 737 gmain polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.679 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.680 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.681 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.682 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 740 gdbus polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.684 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.685 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.686 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.687 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 742 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.688 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.689 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.690 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.691 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.692 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 743 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.693 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.694 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.696 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 744 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.697 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.698 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.700 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.701 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 745 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.702 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.703 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.704 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.705 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.706 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 746 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.707 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.708 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.709 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.710 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.711 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 747 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.712 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.713 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.714 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.715 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 748 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.716 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.718 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.719 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.720 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 749 JS\x20Hel polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.721 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd txt REG 8,1 145440 49355 /usr/lib/polkit-1/polkitd 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.722 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 29097776 36983 /usr/lib64/polkit-1/libmozjs-78.so 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.723 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 131296 10178 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2024-02-19T19:12:00.724 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 4u a_inode 0,14 0 1076 [eventfd:3] 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 5u a_inode 0,14 0 1076 [eventfd:4] 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 6u unix 0xffffa09a86e4d500 0t0 615 type=STREAM (CONNECTED) 2024-02-19T19:12:00.725 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 7u a_inode 0,14 0 1076 [eventfd:8] 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 9u a_inode 0,14 0 1076 [eventfd:10] 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:polkitd 725 751 polkitd polkitd 10u unix 0xffffa09a883d5500 0t0 616 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root txt REG 8,1 306896 20362 /usr/lib/systemd/systemd-logind 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 597694 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.726 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.727 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.728 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.729 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 1u unix 0xffffa09a884ada00 0t0 16768 type=STREAM (CONNECTED) 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 2u unix 0xffffa09a884ada00 0t0 16768 type=STREAM (CONNECTED) 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 3u unix 0xffffa09cb10bdf00 0t0 16793 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.730 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 4u a_inode 0,14 0 1076 [eventpoll:5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,21] 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 5u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 6u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 7r REG 0,21 4096 22963 /sys/devices/virtual/tty/tty0/active 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 8u netlink 0t0 16804 KOBJECT_UEVENT 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 9r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 10r FIFO 0,24 0t0 1168 /run/systemd/inhibit/1.ref 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 11u netlink 0t0 16805 KOBJECT_UEVENT 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 12u netlink 0t0 16806 KOBJECT_UEVENT 2024-02-19T19:12:00.731 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 13u netlink 0t0 16807 KOBJECT_UEVENT 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 14u unix 0xffffa09a884ae900 0t0 16808 type=STREAM (CONNECTED) 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 15u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 16u CHR 13,65 0t0 138 /dev/input/event1 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 17u CHR 13,64 0t0 137 /dev/input/event0 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 18u CHR 13,66 0t0 148 /dev/input/event2 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 19u CHR 4,6 0t0 25 /dev/tty6 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 20r FIFO 0,24 0t0 1667 /run/systemd/sessions/4.ref 2024-02-19T19:12:00.732 INFO:teuthology.orchestra.run.smithi086.stdout:systemd-l 728 root 21r FIFO 0,24 0t0 6603 /run/systemd/sessions/19.ref 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.733 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.734 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.735 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root 1u unix 0xffffa09a88656400 0t0 20800 type=STREAM (CONNECTED) 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root 2u unix 0xffffa09a88656400 0t0 20800 type=STREAM (CONNECTED) 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root 3u IPv4 20810 0t0 TCP *:ssh (LISTEN) 2024-02-19T19:12:00.736 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 777 root 4u IPv6 20812 0t0 TCP *:ssh (LISTEN) 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root txt REG 8,1 15448 10498 /usr/bin/python3.9 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 130464 38212 /usr/lib64/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 107736 38215 /usr/lib64/python3.9/lib-dynload/_datetime.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 204696 37013 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.737 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 143200 10195 /usr/lib64/libgirepository-1.0.so.1.0.0 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 357112 39324 /usr/lib64/python3.9/site-packages/gi/_gi.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.738 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 71408 38249 /usr/lib64/python3.9/lib-dynload/pyexpat.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.739 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 171376 39203 /usr/lib64/python3.9/site-packages/_dbus_bindings.so 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 24736 39204 /usr/lib64/python3.9/site-packages/_dbus_glib_bindings.so 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 23800 38230 /usr/lib64/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.740 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 46560 38224 /usr/lib64/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 76792 9701 /usr/lib64/libbz2.so.1.0.8 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 29224 41980 /usr/lib64/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 41752 38253 /usr/lib64/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 50232 41979 /usr/lib64/python3.9/lib-dynload/_blake2.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 20000 42001 /usr/lib64/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 69048 38220 /usr/lib64/python3.9/lib-dynload/_hashlib.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 24160 41992 /usr/lib64/python3.9/lib-dynload/_random.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.741 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 124184 41989 /usr/lib64/python3.9/lib-dynload/_pickle.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 56424 38237 /usr/lib64/python3.9/lib-dynload/_struct.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 68000 38241 /usr/lib64/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 67528 42002 /usr/lib64/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 23832 38245 /usr/lib64/python3.9/lib-dynload/fcntl.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 15984 38200 /usr/lib64/python3.9/lib-dynload/_bisect.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 15560 38227 /usr/lib64/python3.9/lib-dynload/_opcode.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.742 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 28056 41985 /usr/lib64/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 37592 38251 /usr/lib64/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 101208 41993 /usr/lib64/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 3486240 13688 /usr/lib64/libpython3.9.so.1.0 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.743 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 20528 41991 /usr/lib64/python3.9/lib-dynload/_queue.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 1u unix 0xffffa09a81d52d00 0t0 23206 type=STREAM (CONNECTED) 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 2u unix 0xffffa09a81d52d00 0t0 23206 type=STREAM (CONNECTED) 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 3w REG 8,1 10371 1216 /var/log/tuned/tuned.log 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 4u sock 0,8 0t0 19014 protocol: NETLINK 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 5u unix 0xffffa09a8874ee00 0t0 19030 type=STREAM (CONNECTED) 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 6u a_inode 0,14 0 1076 [eventfd:17] 2024-02-19T19:12:00.744 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 7u unix 0xffffa09a81d54b00 0t0 23269 type=STREAM (CONNECTED) 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 root 8u unix 0xffffa09a81d54600 0t0 23270 type=STREAM (CONNECTED) 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root txt REG 8,1 15448 10498 /usr/bin/python3.9 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 200144 10124 /usr/lib64/libudev.so.1.7.5 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 130464 38212 /usr/lib64/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 107736 38215 /usr/lib64/python3.9/lib-dynload/_datetime.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.745 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 204696 37013 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 1932656 10165 /usr/lib64/libgio-2.0.so.0.6800.4 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 19480 10169 /usr/lib64/libgmodule-2.0.so.0.6800.4 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 375136 10171 /usr/lib64/libgobject-2.0.so.0.6800.4 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 143200 10195 /usr/lib64/libgirepository-1.0.so.1.0.0 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 357112 39324 /usr/lib64/python3.9/site-packages/gi/_gi.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 197952 9776 /usr/lib64/libexpat.so.1.8.10 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 71408 38249 /usr/lib64/python3.9/lib-dynload/pyexpat.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 343752 10128 /usr/lib64/libdbus-1.so.3.19.13 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 171376 39203 /usr/lib64/python3.9/site-packages/_dbus_bindings.so 2024-02-19T19:12:00.748 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 24736 39204 /usr/lib64/python3.9/site-packages/_dbus_glib_bindings.so 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 23800 38230 /usr/lib64/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 46560 38224 /usr/lib64/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 76792 9701 /usr/lib64/libbz2.so.1.0.8 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 29224 41980 /usr/lib64/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 41752 38253 /usr/lib64/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.749 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 50232 41979 /usr/lib64/python3.9/lib-dynload/_blake2.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 20000 42001 /usr/lib64/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 69048 38220 /usr/lib64/python3.9/lib-dynload/_hashlib.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 24160 41992 /usr/lib64/python3.9/lib-dynload/_random.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 124184 41989 /usr/lib64/python3.9/lib-dynload/_pickle.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 56424 38237 /usr/lib64/python3.9/lib-dynload/_struct.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 68000 38241 /usr/lib64/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.750 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 67528 42002 /usr/lib64/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 23832 38245 /usr/lib64/python3.9/lib-dynload/fcntl.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 15984 38200 /usr/lib64/python3.9/lib-dynload/_bisect.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 15560 38227 /usr/lib64/python3.9/lib-dynload/_opcode.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 28056 41985 /usr/lib64/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 37592 38251 /usr/lib64/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 101208 41993 /usr/lib64/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 3486240 13688 /usr/lib64/libpython3.9.so.1.0 2024-02-19T19:12:00.751 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 20528 41991 /usr/lib64/python3.9/lib-dynload/_queue.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 1u unix 0xffffa09a81d52d00 0t0 23206 type=STREAM (CONNECTED) 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 2u unix 0xffffa09a81d52d00 0t0 23206 type=STREAM (CONNECTED) 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 3w REG 8,1 10371 1216 /var/log/tuned/tuned.log 2024-02-19T19:12:00.752 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 4u sock 0,8 0t0 19014 protocol: NETLINK 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 5u unix 0xffffa09a8874ee00 0t0 19030 type=STREAM (CONNECTED) 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 6u a_inode 0,14 0 1076 [eventfd:17] 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 7u unix 0xffffa09a81d54b00 0t0 23269 type=STREAM (CONNECTED) 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:tuned 782 867 tuned root 8u unix 0xffffa09a81d54600 0t0 23270 type=STREAM (CONNECTED) 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.753 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.754 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.755 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.756 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.757 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 1u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 2u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 3u unix 0xffffa09a883d4600 0t0 659 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 4r FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 5u a_inode 0,14 0 1076 [eventpoll:4,6,7,8] 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 6u a_inode 0,14 0 1076 [eventfd:16] 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 7u unix 0xffffa09a86e4b700 0t0 660 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.758 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 8u unix 0xffffa09a86e4e900 0t0 661 /run/gssproxy.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 root 9w FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.759 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.760 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.761 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.762 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.763 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 1u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 2u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 3u unix 0xffffa09a883d4600 0t0 659 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 4r FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 5u a_inode 0,14 0 1076 [eventpoll:4,6,7,8] 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 6u a_inode 0,14 0 1076 [eventfd:16] 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 7u unix 0xffffa09a86e4b700 0t0 660 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.764 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 8u unix 0xffffa09a86e4e900 0t0 661 /run/gssproxy.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 787 gssproxy root 9w FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.765 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.766 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.767 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.768 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.769 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 1u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 2u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 3u unix 0xffffa09a883d4600 0t0 659 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 4r FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 5u a_inode 0,14 0 1076 [eventpoll:4,6,7,8] 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 6u a_inode 0,14 0 1076 [eventfd:16] 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 7u unix 0xffffa09a86e4b700 0t0 660 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 8u unix 0xffffa09a86e4e900 0t0 661 /run/gssproxy.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.770 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 788 gssproxy root 9w FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.771 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.772 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.773 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.774 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 1u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.775 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 2u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 3u unix 0xffffa09a883d4600 0t0 659 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 4r FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 5u a_inode 0,14 0 1076 [eventpoll:4,6,7,8] 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 6u a_inode 0,14 0 1076 [eventfd:16] 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 7u unix 0xffffa09a86e4b700 0t0 660 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 8u unix 0xffffa09a86e4e900 0t0 661 /run/gssproxy.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.776 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 789 gssproxy root 9w FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.777 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.778 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.779 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-02-19T19:12:00.780 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.781 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 1u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 2u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 3u unix 0xffffa09a883d4600 0t0 659 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 4r FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 5u a_inode 0,14 0 1076 [eventpoll:4,6,7,8] 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 6u a_inode 0,14 0 1076 [eventfd:16] 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 7u unix 0xffffa09a86e4b700 0t0 660 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 8u unix 0xffffa09a86e4e900 0t0 661 /run/gssproxy.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 790 gssproxy root 9w FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.782 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root txt REG 8,1 127680 15021 /usr/sbin/gssproxy 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.783 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 70960 15011 /usr/lib64/libev.so.4.0.0 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 16168 15018 /usr/lib64/libverto-libev.so.1.0.0 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 15912 9764 /usr/lib64/libbasicobjects.so.0.1.0 2024-02-19T19:12:00.784 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 20304 9968 /usr/lib64/libpath_utils.so.1.0.1 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 58392 9766 /usr/lib64/libcollection.so.4.1.1 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 142168 11395 /usr/lib64/libgssrpc.so.4.2 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.785 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 16056 9768 /usr/lib64/libref_array.so.1.2.1 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 120640 9970 /usr/lib64/libini_config.so.5.2.1 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 28472 11392 /usr/lib64/libverto.so.1.0.0 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 58368 9689 /usr/lib64/libpopt.so.0.0.1 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 1u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.787 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 2u unix 0xffffa09a887e0f00 0t0 21719 type=STREAM (CONNECTED) 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 3u unix 0xffffa09a883d4600 0t0 659 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 4r FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 5u a_inode 0,14 0 1076 [eventpoll:4,6,7,8] 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 6u a_inode 0,14 0 1076 [eventfd:16] 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 7u unix 0xffffa09a86e4b700 0t0 660 /var/lib/gssproxy/default.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 8u unix 0xffffa09a86e4e900 0t0 661 /run/gssproxy.sock type=STREAM (LISTEN) 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:gssproxy 786 791 gssproxy root 9w FIFO 0,13 0t0 664 pipe 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root cwd DIR 8,1 4096 1657 /var/spool/at 2024-02-19T19:12:00.788 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root txt REG 8,1 32680 12473 /usr/sbin/atd 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.789 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:atd 793 root 3uW REG 0,24 4 1191 /run/atd.pid 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.790 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root txt REG 8,1 74552 11234 /usr/sbin/crond 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.791 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.792 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root 1u unix 0xffffa09a8874b200 0t0 19002 type=STREAM (CONNECTED) 2024-02-19T19:12:00.793 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root 2u unix 0xffffa09a8874b200 0t0 19002 type=STREAM (CONNECTED) 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root 3uW REG 0,24 4 1190 /run/crond.pid 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root 4u unix 0xffffa09a88758f00 0t0 19005 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:crond 794 root 5r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe txt REG 8,1 80984 12959 /usr/sbin/nrpe 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.794 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe 3u IPv4 16915 0t0 TCP *:nrpe (LISTEN) 2024-02-19T19:12:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe 4u unix 0xffffa09cb10bf300 0t0 16907 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:nrpe 886 nrpe 5u IPv6 16916 0t0 TCP *:nrpe (LISTEN) 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root txt REG 8,1 58072 13690 /usr/sbin/agetty 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.796 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root 0u CHR 4,1 0t0 20 /dev/tty1 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root 1u CHR 4,1 0t0 20 /dev/tty1 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root 2u CHR 4,1 0t0 20 /dev/tty1 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1325 root 4r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root txt REG 8,1 58072 13690 /usr/sbin/agetty 2024-02-19T19:12:00.797 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.798 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root 0u CHR 4,65 0t0 114 /dev/ttyS1 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root 1u CHR 4,65 0t0 114 /dev/ttyS1 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root 2u CHR 4,65 0t0 114 /dev/ttyS1 2024-02-19T19:12:00.799 INFO:teuthology.orchestra.run.smithi086.stdout:agetty 1328 root 4r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu DEL REG 8,1 1167 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.800 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu DEL REG 8,1 1440 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.801 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.802 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 1u unix 0xffffa09a884acb00 0t0 17040 type=STREAM (CONNECTED) 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 2u unix 0xffffa09a884acb00 0t0 17040 type=STREAM (CONNECTED) 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 3u unix 0xffffa09cb10bb200 0t0 17069 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 4u a_inode 0,14 0 1076 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-02-19T19:12:00.804 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 5u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 6r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 7r DIR 0,25 0 3392 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 8u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 9u a_inode 0,14 0 1076 [eventpoll:10,13] 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 10r REG 0,20 0 17058 /proc/1391/mountinfo 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 11r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 12u unix 0xffffa09a887e7300 0t0 21857 /run/user/1000/bus type=STREAM (LISTEN) 2024-02-19T19:12:00.805 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 13r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 14r REG 0,20 0 4026532076 /proc/swaps 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 15u netlink 0t0 17079 KOBJECT_UEVENT 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 16u unix 0xffffa09cb10bc600 0t0 17086 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 17u unix 0xffffa09cb10bd500 0t0 17087 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 18u unix 0xffffa09cb10b9900 0t0 17088 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 19u unix 0xffffa09a884a9e00 0t0 17089 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 20u unix 0xffffa09a884ab700 0t0 17091 type=STREAM (CONNECTED) 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 1391 ubuntu 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.806 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 1403 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 1403 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 1403 root txt unknown /proc/1403/exe 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu DEL REG 8,1 63926 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.808 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:00.809 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.810 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.811 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu DEL REG 8,1 63928 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.812 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 1u unix 0xffffa09a884acb00 0t0 17040 type=STREAM (CONNECTED) 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 2u unix 0xffffa09a884acb00 0t0 17040 type=STREAM (CONNECTED) 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 3u a_inode 0,14 0 1076 [eventfd:13] 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 4u a_inode 0,14 0 1076 [eventfd:14] 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 6w FIFO 0,13 0t0 17045 pipe 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 1416 ubuntu 7u unix 0xffffa09cb10b8000 0t0 17061 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.813 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root txt REG 8,1 769928 12237 /usr/sbin/rsyslogd 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 33592 12207 /usr/lib64/rsyslog/imjournal.so 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 42248 12214 /usr/lib64/rsyslog/imuxsock.so 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 32808 12215 /usr/lib64/rsyslog/lmnet.so 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.814 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 36328 9712 /usr/lib64/libuuid.so.1.3.0 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 52856 12201 /usr/lib64/libfastjson.so.4.3.0 2024-02-19T19:12:00.815 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 16032 12200 /usr/lib64/libestr.so.0.0.0 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 3r CHR 1,9 0t0 9 /dev/urandom 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 4u unix 0xffffa09a8c990000 0t0 22062 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.816 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 5w REG 8,1 627410 1642 /var/log/secure 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 6r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 7w REG 8,1 1734 1220 /var/log/cron 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 8w REG 8,1 3666739 1641 /var/log/messages 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 9r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 root 10r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.817 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root txt REG 8,1 769928 12237 /usr/sbin/rsyslogd 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 33592 12207 /usr/lib64/rsyslog/imjournal.so 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 42248 12214 /usr/lib64/rsyslog/imuxsock.so 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 32808 12215 /usr/lib64/rsyslog/lmnet.so 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.818 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 36328 9712 /usr/lib64/libuuid.so.1.3.0 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.819 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 52856 12201 /usr/lib64/libfastjson.so.4.3.0 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 16032 12200 /usr/lib64/libestr.so.0.0.0 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 3r CHR 1,9 0t0 9 /dev/urandom 2024-02-19T19:12:00.820 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 4u unix 0xffffa09a8c990000 0t0 22062 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 5w REG 8,1 627410 1642 /var/log/secure 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 6r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 7w REG 8,1 1734 1220 /var/log/cron 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 8w REG 8,1 3666739 1641 /var/log/messages 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 9r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3617 in:imjour root 10r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root txt REG 8,1 769928 12237 /usr/sbin/rsyslogd 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 33592 12207 /usr/lib64/rsyslog/imjournal.so 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 42248 12214 /usr/lib64/rsyslog/imuxsock.so 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 32808 12215 /usr/lib64/rsyslog/lmnet.so 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.822 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 36328 9712 /usr/lib64/libuuid.so.1.3.0 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 52856 12201 /usr/lib64/libfastjson.so.4.3.0 2024-02-19T19:12:00.823 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 16032 12200 /usr/lib64/libestr.so.0.0.0 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 3r CHR 1,9 0t0 9 /dev/urandom 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 4u unix 0xffffa09a8c990000 0t0 22062 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.824 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 5w REG 8,1 627410 1642 /var/log/secure 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 6r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 7w REG 8,1 1734 1220 /var/log/cron 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 8w REG 8,1 3666739 1641 /var/log/messages 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 9r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:rsyslogd 3604 3618 rs:main root 10r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:tls-strp 15814 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:tls-strp 15814 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:tls-strp 15814 root txt unknown /proc/15814/exe 2024-02-19T19:12:00.825 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root txt REG 8,1 630408 11631 /usr/sbin/smartd 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.826 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 2306456 9691 /usr/lib64/libstdc++.so.6.0.29 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.827 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:smartd 15862 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21195 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21195 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21195 root txt unknown /proc/21195/exe 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21224 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21224 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21224 root txt unknown /proc/21224/exe 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21253 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21253 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21253 root txt unknown /proc/21253/exe 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21282 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21282 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21282 root txt unknown /proc/21282/exe 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21311 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21311 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.829 INFO:teuthology.orchestra.run.smithi086.stdout:kdmflush/ 21311 root txt unknown /proc/21311/exe 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfsalloc 21475 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfsalloc 21475 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfsalloc 21475 root txt unknown /proc/21475/exe 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfs_mru_c 21476 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfs_mru_c 21476 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfs_mru_c 21476 root txt unknown /proc/21476/exe 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-buf/d 21477 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-buf/d 21477 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.830 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-buf/d 21477 root txt unknown /proc/21477/exe 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-conv/ 21478 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-conv/ 21478 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-conv/ 21478 root txt unknown /proc/21478/exe 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-recla 21479 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-recla 21479 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-recla 21479 root txt unknown /proc/21479/exe 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-block 21480 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-block 21480 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.831 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-block 21480 root txt unknown /proc/21480/exe 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-inode 21481 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-inode 21481 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-inode 21481 root txt unknown /proc/21481/exe 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-log/d 21482 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-log/d 21482 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-log/d 21482 root txt unknown /proc/21482/exe 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-cil/d 21483 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.832 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-cil/d 21483 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:xfs-cil/d 21483 root txt unknown /proc/21483/exe 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:xfsaild/d 21484 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:xfsaild/d 21484 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:xfsaild/d 21484 root txt unknown /proc/21484/exe 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:00.833 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 19520 32283 /usr/lib64/security/pam_lastlog.so 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15264 32290 /usr/lib64/security/pam_motd.so 2024-02-19T19:12:00.834 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15232 32292 /usr/lib64/security/pam_nologin.so 2024-02-19T19:12:00.835 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 19368 32301 /usr/lib64/security/pam_sepermit.so 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.836 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.837 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.838 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.839 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 3u unix 0xffffa09cb10ba300 0t0 91962 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 4u IPv4 85714 0t0 TCP smithi086.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:38608 (ESTABLISHED) 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 6u unix 0xffffa09a86ebf800 0t0 91937 type=STREAM (CONNECTED) 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 7w FIFO 0,24 0t0 1667 /run/systemd/sessions/4.ref 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23404 root 8u unix 0xffffa09a86ebc100 0t0 91966 type=STREAM (CONNECTED) 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:00.840 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 19520 32283 /usr/lib64/security/pam_lastlog.so 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:00.841 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15264 32290 /usr/lib64/security/pam_motd.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:00.842 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15232 32292 /usr/lib64/security/pam_nologin.so 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 19368 32301 /usr/lib64/security/pam_sepermit.so 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.843 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.845 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:00.846 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 3u unix 0xffffa09cb10ba300 0t0 91962 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 4u IPv4 85714 0t0 TCP smithi086.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:38608 (ESTABLISHED) 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 5u unix 0xffffa09a86ebd500 0t0 91965 type=STREAM (CONNECTED) 2024-02-19T19:12:00.847 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 6u unix 0xffffa09a86ebf800 0t0 91937 type=STREAM (CONNECTED) 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 7w FIFO 0,24 0t0 1667 /run/systemd/sessions/4.ref 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 12r FIFO 0,13 0t0 113046 pipe 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 13r FIFO 0,13 0t0 149999 pipe 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 14r FIFO 0,13 0t0 113047 pipe 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 15r FIFO 0,13 0t0 157222 pipe 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 16r FIFO 0,13 0t0 150000 pipe 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 17r FIFO 0,13 0t0 164826 pipe 2024-02-19T19:12:00.848 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 18r FIFO 0,13 0t0 157223 pipe 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 19r FIFO 0,13 0t0 247763 pipe 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 20r FIFO 0,13 0t0 164827 pipe 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 21r FIFO 0,13 0t0 254676 pipe 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 22r FIFO 0,13 0t0 247764 pipe 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 23408 ubuntu 24r FIFO 0,13 0t0 254677 pipe 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.849 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony txt REG 8,1 353408 12490 /usr/sbin/chronyd 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 1591920 9734 /usr/lib64/libunistring.so.2.1.0 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 626152 9989 /usr/lib64/libhogweed.so.6.5 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 2291576 9994 /usr/lib64/libgnutls.so.30.33.1 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 340456 9991 /usr/lib64/libnettle.so.8.5 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 94112 9901 /usr/lib64/libtasn1.so.6.6.0 2024-02-19T19:12:00.850 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 130952 9772 /usr/lib64/libidn2.so.0.3.7 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.851 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 3u unix 0xffffa09a88759900 0t0 95126 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 5u IPv4 95131 0t0 UDP localhost:323 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 6u IPv6 95132 0t0 UDP localhost:323 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:chronyd 23469 chrony 7u unix 0xffffa09a8875d500 0t0 95133 /run/chrony/chronyd.sock type=DGRAM (CONNECTED) 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph txt REG 8,1 15448 10498 /usr/bin/python3.9 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 46560 38224 /usr/lib64/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 76792 9701 /usr/lib64/libbz2.so.1.0.8 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 29224 41980 /usr/lib64/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 41752 38253 /usr/lib64/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 68000 38241 /usr/lib64/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 101208 41993 /usr/lib64/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 23800 38230 /usr/lib64/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 37592 38251 /usr/lib64/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 67528 42002 /usr/lib64/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 3486240 13688 /usr/lib64/libpython3.9.so.1.0 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 20000 42001 /usr/lib64/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 28056 41985 /usr/lib64/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph 1u unix 0xffffa09a84b27800 0t0 95687 type=STREAM (CONNECTED) 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 23700 ceph 2u unix 0xffffa09a84b27800 0t0 95687 type=STREAM (CONNECTED) 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 25643 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 25643 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 25643 root txt unknown /proc/25643/exe 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 25908 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 25908 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 25908 root txt unknown /proc/25908/exe 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 26367 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.855 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 26367 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 26367 root txt unknown /proc/26367/exe 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 597694 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.856 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.857 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.858 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.859 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 1u unix 0xffffa09a88657800 0t0 108618 type=STREAM (CONNECTED) 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 2u unix 0xffffa09a88657800 0t0 108618 type=STREAM (CONNECTED) 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 3u unix 0xffffa09a88ebe400 0t0 108649 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 4u a_inode 0,14 0 1076 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 5u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.860 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 6r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 7r DIR 0,25 0 9090 /sys/fs/cgroup/user.slice/user-0.slice/user@0.service 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 8u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 9u a_inode 0,14 0 1076 [eventpoll:10,13] 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 10r REG 0,20 0 108636 /proc/28296/mountinfo 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 11r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 12u unix 0xffffa09a88652d00 0t0 108677 /run/user/0/bus type=STREAM (LISTEN) 2024-02-19T19:12:00.861 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 13r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 14r REG 0,20 0 4026532076 /proc/swaps 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 15u netlink 0t0 108659 KOBJECT_UEVENT 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 16u unix 0xffffa09a88ebe900 0t0 108664 /run/user/0/systemd/notify type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 17u unix 0xffffa09a88ebf800 0t0 108665 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 18u unix 0xffffa09a88eb8500 0t0 108666 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 19u unix 0xffffa09a88651900 0t0 108667 /run/user/0/systemd/private type=STREAM (LISTEN) 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 20u unix 0xffffa09a88655000 0t0 108669 type=STREAM (CONNECTED) 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:systemd 28296 root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.862 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 597694 1143 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.863 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-02-19T19:12:00.864 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 21219 1168 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.865 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-02-19T19:12:00.867 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 1u unix 0xffffa09a88657800 0t0 108618 type=STREAM (CONNECTED) 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 2u unix 0xffffa09a88657800 0t0 108618 type=STREAM (CONNECTED) 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 3u a_inode 0,14 0 1076 [eventfd:9] 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 4u a_inode 0,14 0 1076 [eventfd:11] 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 6w FIFO 0,13 0t0 108620 pipe 2024-02-19T19:12:00.868 INFO:teuthology.orchestra.run.smithi086.stdout:(sd-pam) 28299 root 7u unix 0xffffa09a88ebdf00 0t0 108639 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root txt REG 8,1 185296 11759 /usr/bin/sudo 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.869 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:00.870 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:00.871 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:00.872 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 472568 54513 /usr/libexec/sudo/sudoers.so 2024-02-19T19:12:00.873 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.874 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 119832 54509 /usr/libexec/sudo/libsudo_util.so.0.0.0 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.875 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 0r FIFO 0,13 0t0 113045 pipe 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 1w FIFO 0,13 0t0 113046 pipe 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 2w FIFO 0,13 0t0 113047 pipe 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 3r FIFO 0,13 0t0 111833 pipe 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 4w FIFO 0,13 0t0 111833 pipe 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 5u netlink 0t0 111838 AUDIT 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 28830 root 6u unix 0xffffa09a8c990f00 0t0 111839 type=DGRAM (CONNECTED) 2024-02-19T19:12:00.876 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root txt REG 8,1 90216 11090 /usr/bin/journalctl 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.877 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:00.878 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:00.879 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.880 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.881 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root 0r FIFO 0,13 0t0 113045 pipe 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root 1w FIFO 0,13 0t0 113046 pipe 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root 2w FIFO 0,13 0t0 113047 pipe 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root 5r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 28863 root 6r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:00.882 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 29543 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 29543 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 29543 root txt unknown /proc/29543/exe 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 29684 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 29684 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 29684 root txt unknown /proc/29684/exe 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 30681 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 30681 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 30681 root txt unknown /proc/30681/exe 2024-02-19T19:12:00.883 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 30688 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 30688 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 30688 root txt unknown /proc/30688/exe 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.885 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.886 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.887 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 3u unix 0xffffa09a8875f300 0t0 123691 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 5u a_inode 0,14 0 1076 [eventfd:18] 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 7r FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 9r FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 10u unix 0xffffa09a88eb9e00 0t0 128020 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.888 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 11r FIFO 8,1 0t0 528460 /var/lib/containers/storage/overlay-containers/3ad8b6c1f8d2fc835032cbf1f6a16119a3255e52fc3b474d5b672229b6aa1208/userdata/ctl 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 12u unix 0xffffa09a88ebc100 0t0 128013 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 13w FIFO 8,1 0t0 528460 /var/lib/containers/storage/overlay-containers/3ad8b6c1f8d2fc835032cbf1f6a16119a3255e52fc3b474d5b672229b6aa1208/userdata/ctl 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 14r FIFO 8,1 0t0 528461 /var/lib/containers/storage/overlay-containers/3ad8b6c1f8d2fc835032cbf1f6a16119a3255e52fc3b474d5b672229b6aa1208/userdata/winsz 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 15w FIFO 8,1 0t0 528461 /var/lib/containers/storage/overlay-containers/3ad8b6c1f8d2fc835032cbf1f6a16119a3255e52fc3b474d5b672229b6aa1208/userdata/winsz 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 35700 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.889 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 35702 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35720 log root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35722 msgr-work root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35723 msgr-work root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35724 msgr-work root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35728 service root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35729 admin_soc root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root cwd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root rtd DIR 0,49 4096 528445 / 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root txt REG 0,49 1429736 410528 /usr/bin/ceph-exporter 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 410528 /usr/bin/ceph-exporter (stat: No such file or directory) 2024-02-19T19:12:00.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 421002 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 421000 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420998 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420831 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420863 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420871 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:00.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420986 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420927 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420786 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 420358 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 1w FIFO 0,13 0t0 128011 pipe 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 2w FIFO 0,13 0t0 128012 pipe 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 3u a_inode 0,14 0 1076 [eventpoll:4] 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 4r FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 5w FIFO 0,13 0t0 122856 pipe 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 6u a_inode 0,14 0 1076 [eventpoll:7] 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 7r FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 8w FIFO 0,13 0t0 122857 pipe 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 9u a_inode 0,14 0 1076 [eventpoll:10] 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 10r FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 11w FIFO 0,13 0t0 122858 pipe 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 12w REG 8,1 101380 528424 /var/log/ceph/ceph-client.ceph-exporter.smithi086.log 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 13r FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 14w FIFO 0,13 0t0 122859 pipe 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 15u unix 0xffffa09a84b24100 0t0 122860 /var/run/ceph/ceph-client.ceph-exporter.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 16u a_inode 0,14 0 1076 [eventfd:12] 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 17u a_inode 0,14 0 1076 [eventpoll:16,18] 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 18u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 19u a_inode 0,14 0 1076 [eventfd:15] 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 20u a_inode 0,14 0 1076 [eventpoll:19,21,23] 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 21u a_inode 0,14 0 1076 [timerfd] 2024-02-19T19:12:00.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-expo 35719 35730 ceph-expo root 23u IPv4 125476 0t0 TCP *:9926 (LISTEN) 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:00.953 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:00.954 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:00.955 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 3u unix 0xffffa09a8c993c00 0t0 125916 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.956 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 5u a_inode 0,14 0 1076 [eventfd:26] 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 7r FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 9r FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 10u unix 0xffffa09a8875da00 0t0 131219 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 11r FIFO 8,1 0t0 530481 /var/lib/containers/storage/overlay-containers/19bceea5d35c0d566bcf2b94ae0de15441b48facff2e36c0e4619bb5e19da02b/userdata/ctl 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 12u unix 0xffffa09a8875a300 0t0 131202 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 13w FIFO 8,1 0t0 530481 /var/lib/containers/storage/overlay-containers/19bceea5d35c0d566bcf2b94ae0de15441b48facff2e36c0e4619bb5e19da02b/userdata/ctl 2024-02-19T19:12:00.957 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 14r FIFO 8,1 0t0 530482 /var/lib/containers/storage/overlay-containers/19bceea5d35c0d566bcf2b94ae0de15441b48facff2e36c0e4619bb5e19da02b/userdata/winsz 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 15w FIFO 8,1 0t0 530482 /var/lib/containers/storage/overlay-containers/19bceea5d35c0d566bcf2b94ae0de15441b48facff2e36c0e4619bb5e19da02b/userdata/winsz 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 36643 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 36645 nobody 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:00.958 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:00.959 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:00.959 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody txt REG 0,71 19779640 528420 /bin/node_exporter 2024-02-19T19:12:00.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:00 smithi039 ceph-mon[92224]: from='client.? 172.21.15.39:0/661215398' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T19:12:00.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:00 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:12:00.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:00 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:12:00.960 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:00 smithi039 ceph-mon[92224]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody mem REG 8,1 528420 /bin/node_exporter (stat: No such file or directory) 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 3u IPv6 125932 0t0 TCP *:jetdirect (LISTEN) 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 4u a_inode 0,14 0 1076 [eventpoll:3,5,7] 2024-02-19T19:12:00.999 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 5r FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 6w FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 nobody 7u IPv6 202272 0t0 TCP smithi086.front.sepia.ceph.com:jetdirect->smithi039.front.sepia.ceph.com:33902 (ESTABLISHED) 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody txt REG 0,71 19779640 528420 /bin/node_exporter 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody mem REG 8,1 528420 /bin/node_exporter (stat: No such file or directory) 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:01.000 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 3u IPv6 125932 0t0 TCP *:jetdirect (LISTEN) 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 4u a_inode 0,14 0 1076 [eventpoll:3,5,7] 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 5r FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 6w FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36650 node_expo nobody 7u IPv6 202272 0t0 TCP smithi086.front.sepia.ceph.com:jetdirect->smithi039.front.sepia.ceph.com:33902 (ESTABLISHED) 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody txt REG 0,71 19779640 528420 /bin/node_exporter 2024-02-19T19:12:01.001 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody mem REG 8,1 528420 /bin/node_exporter (stat: No such file or directory) 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 3u IPv6 125932 0t0 TCP *:jetdirect (LISTEN) 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 4u a_inode 0,14 0 1076 [eventpoll:3,5,7] 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 5r FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.002 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 6w FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36651 node_expo nobody 7u IPv6 202272 0t0 TCP smithi086.front.sepia.ceph.com:jetdirect->smithi039.front.sepia.ceph.com:33902 (ESTABLISHED) 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody txt REG 0,71 19779640 528420 /bin/node_exporter 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody mem REG 8,1 528420 /bin/node_exporter (stat: No such file or directory) 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:01.003 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 3u IPv6 125932 0t0 TCP *:jetdirect (LISTEN) 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 4u a_inode 0,14 0 1076 [eventpoll:3,5,7] 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 5r FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 6w FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36652 node_expo nobody 7u IPv6 202272 0t0 TCP smithi086.front.sepia.ceph.com:jetdirect->smithi039.front.sepia.ceph.com:33902 (ESTABLISHED) 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody txt REG 0,71 19779640 528420 /bin/node_exporter 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody mem REG 8,1 528420 /bin/node_exporter (stat: No such file or directory) 2024-02-19T19:12:01.004 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 3u IPv6 125932 0t0 TCP *:jetdirect (LISTEN) 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 4u a_inode 0,14 0 1076 [eventpoll:3,5,7] 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 5r FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 6w FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 36653 node_expo nobody 7u IPv6 202272 0t0 TCP smithi086.front.sepia.ceph.com:jetdirect->smithi039.front.sepia.ceph.com:33902 (ESTABLISHED) 2024-02-19T19:12:01.005 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody cwd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody rtd DIR 0,71 4096 528412 / 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody txt REG 0,71 19779640 528420 /bin/node_exporter 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody mem REG 8,1 528420 /bin/node_exporter (stat: No such file or directory) 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 1w FIFO 0,13 0t0 131200 pipe 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 2w FIFO 0,13 0t0 131201 pipe 2024-02-19T19:12:01.006 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 3u IPv6 125932 0t0 TCP *:jetdirect (LISTEN) 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 4u a_inode 0,14 0 1076 [eventpoll:3,5,7] 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 5r FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 6w FIFO 0,13 0t0 125928 pipe 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:node_expo 36649 39953 node_expo nobody 7u IPv6 202272 0t0 TCP smithi086.front.sepia.ceph.com:jetdirect->smithi039.front.sepia.ceph.com:33902 (ESTABLISHED) 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 38819 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 38819 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 38819 root txt unknown /proc/38819/exe 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 42694 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.007 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 42694 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 42694 root txt unknown /proc/42694/exe 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root txt REG 8,1 185296 11759 /usr/bin/sudo 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.008 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:01.009 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:01.010 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:01.011 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 472568 54513 /usr/libexec/sudo/sudoers.so 2024-02-19T19:12:01.012 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.013 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 119832 54509 /usr/libexec/sudo/libsudo_util.so.0.0.0 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 0r FIFO 0,13 0t0 149998 pipe 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 1w FIFO 0,13 0t0 149999 pipe 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 2w FIFO 0,13 0t0 150000 pipe 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 3r FIFO 0,13 0t0 150790 pipe 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 4w FIFO 0,13 0t0 150790 pipe 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 5u netlink 0t0 150795 AUDIT 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 44926 root 6u unix 0xffffa09a88eb8f00 0t0 150796 type=DGRAM (CONNECTED) 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.015 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root txt REG 8,1 90216 11090 /usr/bin/journalctl 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.016 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.017 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.018 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.019 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.020 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root 0r FIFO 0,13 0t0 149998 pipe 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root 1w FIFO 0,13 0t0 149999 pipe 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root 2w FIFO 0,13 0t0 150000 pipe 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root 5r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 44943 root 6r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:01.021 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 49821 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 49821 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 49821 root txt unknown /proc/49821/exe 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root txt REG 8,1 185296 11759 /usr/bin/sudo 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.022 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:01.023 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.024 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:01.025 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 472568 54513 /usr/libexec/sudo/sudoers.so 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.026 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.027 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 119832 54509 /usr/libexec/sudo/libsudo_util.so.0.0.0 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.028 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 0r FIFO 0,13 0t0 157221 pipe 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 1w FIFO 0,13 0t0 157222 pipe 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 2w FIFO 0,13 0t0 157223 pipe 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 3r FIFO 0,13 0t0 157232 pipe 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 4w FIFO 0,13 0t0 157232 pipe 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 5u netlink 0t0 157237 AUDIT 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 51464 root 6u unix 0xffffa09cb10bc100 0t0 157238 type=DGRAM (CONNECTED) 2024-02-19T19:12:01.029 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root txt REG 8,1 90216 11090 /usr/bin/journalctl 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.030 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.031 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:01.032 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:01.033 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.034 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root 0r FIFO 0,13 0t0 157221 pipe 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root 1w FIFO 0,13 0t0 157222 pipe 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root 2w FIFO 0,13 0t0 157223 pipe 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:01.035 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root 5r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 51494 root 6r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root txt REG 8,1 185296 11759 /usr/bin/sudo 2024-02-19T19:12:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:01.036 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:01.037 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:01.038 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 688208 10009 /usr/lib64/libssl.so.3.0.7 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 366424 10411 /usr/lib64/libevent-2.1.so.7.0.1 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:01.039 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 128416 11415 /usr/lib64/libsasl2.so.3.0.0 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 417792 11423 /usr/lib64/libldap_r.so.2.0.200 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.040 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 70400 11419 /usr/lib64/liblber.so.2.0.200 2024-02-19T19:12:01.041 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.041 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:01.041 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 472568 54513 /usr/libexec/sudo/sudoers.so 2024-02-19T19:12:01.041 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.042 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 119832 54509 /usr/libexec/sudo/libsudo_util.so.0.0.0 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.043 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 0r FIFO 0,13 0t0 164825 pipe 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 1w FIFO 0,13 0t0 164826 pipe 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 2w FIFO 0,13 0t0 164827 pipe 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 3r FIFO 0,13 0t0 168194 pipe 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 4w FIFO 0,13 0t0 168194 pipe 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 5u netlink 0t0 168199 AUDIT 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:sudo 57940 root 6u unix 0xffffa09a887c5a00 0t0 168200 type=DGRAM (CONNECTED) 2024-02-19T19:12:01.044 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root cwd DIR 8,1 4096 1026 /home/ubuntu 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root txt REG 8,1 90216 11090 /usr/bin/journalctl 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.045 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.046 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.047 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-02-19T19:12:01.048 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 283152 10156 /usr/lib64/libmount.so.1.1.0 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 229360 10126 /usr/lib64/libblkid.so.1.1.0 2024-02-19T19:12:01.049 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 41120 9806 /usr/lib64/libacl.so.1.1.2301 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root 0r FIFO 0,13 0t0 164825 pipe 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root 1w FIFO 0,13 0t0 164826 pipe 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root 2w FIFO 0,13 0t0 164827 pipe 2024-02-19T19:12:01.050 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root 3r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root 5r REG 0,24 16777216 1957 /run/log/journal/15949e3c88704300882482eddfd7baae/system.journal 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:journalct 57957 root 6r REG 0,24 6383936 58 /run/log/journal/15949e3c88704300882482eddfd7baae/system@48e212dd36874bc198f237ff204fe066-0000000000000001-000611c0774863d0.journal 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root cwd DIR 8,1 4096 1136 /var/spool/anacron 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root txt REG 8,1 41136 11229 /usr/sbin/anacron 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.051 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.052 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 3uW REG 8,1 0 1138 /var/spool/anacron/cron.daily 2024-02-19T19:12:01.053 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 4u unix 0xffffa09bc603ee00 0t0 171485 type=DGRAM (CONNECTED) 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 5uW REG 8,1 0 1139 /var/spool/anacron/cron.weekly 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:anacron 61215 root 6uW REG 8,1 0 1140 /var/spool/anacron/cron.monthly 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 62958 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 62958 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 62958 root txt unknown /proc/62958/exe 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 63236 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 63236 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.054 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 63236 root txt unknown /proc/63236/exe 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-msgr 63237 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-msgr 63237 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-msgr 63237 root txt unknown /proc/63237/exe 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-watc 63241 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-watc 63241 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-watc 63241 root txt unknown /proc/63241/exe 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-comp 63242 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-comp 63242 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-comp 63242 root txt unknown /proc/63242/exe 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 63435 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 63435 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 63435 root txt unknown /proc/63435/exe 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 67698 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 67698 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/0 67698 root txt unknown /proc/67698/exe 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 72155 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 72155 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.056 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 72155 root txt unknown /proc/72155/exe 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 72156 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 72156 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/5 72156 root txt unknown /proc/72156/exe 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 72932 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 72932 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/6 72932 root txt unknown /proc/72932/exe 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.057 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.058 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.059 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.060 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 3u unix 0xffffa09a8c990a00 0t0 200796 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 5u a_inode 0,14 0 1076 [eventfd:21] 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 7r FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.061 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 9r FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 10u unix 0xffffa09bc603d500 0t0 200074 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 11r FIFO 8,1 0t0 426296 /var/lib/containers/storage/overlay-containers/fbd829f78f68a1ac7c84a6804b7101ffa9f0fa0e332890bf9e8c38b25b22075b/userdata/ctl 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 12u unix 0xffffa09bc6039900 0t0 200048 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 13w FIFO 8,1 0t0 426296 /var/lib/containers/storage/overlay-containers/fbd829f78f68a1ac7c84a6804b7101ffa9f0fa0e332890bf9e8c38b25b22075b/userdata/ctl 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 14r FIFO 8,1 0t0 426297 /var/lib/containers/storage/overlay-containers/fbd829f78f68a1ac7c84a6804b7101ffa9f0fa0e332890bf9e8c38b25b22075b/userdata/winsz 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 15w FIFO 8,1 0t0 426297 /var/lib/containers/storage/overlay-containers/fbd829f78f68a1ac7c84a6804b7101ffa9f0fa0e332890bf9e8c38b25b22075b/userdata/winsz 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 74266 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 74268 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77758 log ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77760 msgr-work ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77761 msgr-work ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77762 msgr-work ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77766 service ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77767 admin_soc ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77768 ceph_time ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77769 fn_anonym ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77770 signal_ha ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77771 mgrsb-fin ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77772 ms_dispat ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77773 ms_local ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77774 io_contex ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77775 io_contex ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77776 safe_time ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77777 safe_time ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77778 safe_time ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77779 fn_anonym ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77780 flusher ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77781 safe_time ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77846 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77847 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77848 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77849 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77850 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77851 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77852 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77853 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77854 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77855 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77856 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77857 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77858 prometheu ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77860 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77861 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77862 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77863 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77864 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77865 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77866 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77867 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77868 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77869 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph cwd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph rtd DIR 0,82 4096 658714 / 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph txt REG 0,82 5853720 531019 /usr/bin/ceph-mgr 2024-02-19T19:12:01.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 531019 /usr/bin/ceph-mgr (path inode=2781) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406045 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404108 /usr/lib64/libnettle.so.6.5 (stat: No such file or directory) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 543084 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542946 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542684 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542816 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542818 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542378 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541515 /usr/lib64/libquadmath.so.0.0.0 (path inode=2828) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541370 /usr/lib64/libgfortran.so.5.0.0 (path inode=2832) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541491 /usr/lib64/libopenblas-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541493 /usr/lib64/libopenblasp-r0.3.15.so (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542284 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541622 /usr/lib64/libxmlsec1-openssl.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541423 /usr/lib64/libltdl.so.7.3.1 (stat: No such file or directory) 2024-02-19T19:12:01.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541624 /usr/lib64/libxmlsec1.so.1.2.25 (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 546000 /usr/lib64/python3.6/site-packages/xmlsec.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542067 /usr/lib64/python3.6/site-packages/lxml/_elementpath.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404032 /usr/lib64/libgpg-error.so.0.24.2 (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404014 /usr/lib64/libgcrypt.so.20.2.5 (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404240 /usr/lib64/libxml2.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541353 /usr/lib64/libexslt.so.0.8.20 (path inode=10716) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541627 /usr/lib64/libxslt.so.1.1.32 (stat: No such file or directory) 2024-02-19T19:12:01.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542073 /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406008 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404182 /usr/lib64/libsepol.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404050 /usr/lib64/libjson-c.so.4.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403990 /usr/lib64/libdevmapper.so.1.02 (path inode=11464) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403983 /usr/lib64/libcryptsetup.so.12.6.0 (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541525 /usr/lib64/librbd.so.1.19.0 (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 543028 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404002 /usr/lib64/libexpat.so.1.6.7 (stat: No such file or directory) 2024-02-19T19:12:01.930 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:00 smithi086 ceph-mon[82675]: from='client.? 172.21.15.39:0/661215398' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T19:12:01.930 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:00 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' 2024-02-19T19:12:01.930 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:00 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi039", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:12:01.930 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:00 smithi086 ceph-mon[82675]: from='mgr.34104 172.21.15.39:0/2831038795' entity='mgr.smithi039.ydmpuh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:12:01.935 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:01 smithi039 ceph-mon[92224]: pgmap v212: 65 pgs: 65 active+clean; 171 MiB data, 815 MiB used, 536 GiB / 536 GiB avail; 4.0 MiB/s rd, 5.5 MiB/s wr, 422 op/s 2024-02-19T19:12:01.935 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:01 smithi039 ceph-mon[92224]: Upgrade: Updating ceph-exporter.smithi039 (1/2) 2024-02-19T19:12:01.935 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:01 smithi039 ceph-mon[92224]: Deploying daemon ceph-exporter.smithi039 on smithi039 2024-02-19T19:12:01.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406039 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406001 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404242 /usr/lib64/libyaml-0.so.2.0.5 (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541667 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541833 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 405990 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406017 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406028 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406046 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406019 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 542167 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406007 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406012 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406025 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406030 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406032 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541307 /usr/lib64/libcephfs.so.2.0.0 (path inode=2296) 2024-02-19T19:12:01.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541662 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541683 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541689 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406021 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 405991 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406022 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 405992 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406010 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404079 /usr/lib64/liblzma.so.5.2.4 (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406015 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403966 /usr/lib64/libbz2.so.1.0.6 (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 405993 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406048 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404006 /usr/lib64/libffi.so.6.0.2 (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406002 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406024 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406005 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 543022 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406013 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541191 /usr/lib64/gconv/gconv-modules.cache (path inode=29960) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406018 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541519 /usr/lib64/librados.so.2.0.0 (path inode=1770) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:01.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 404194 /usr/lib64/libsqlite3.so.0.8.6 (path inode=9721) 2024-02-19T19:12:01.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph mem REG 8,1 541308 /usr/lib64/libcephsqlite.so (path inode=2546) 2024-02-19T19:12:01.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:01.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 1w FIFO 0,13 0t0 200046 pipe 2024-02-19T19:12:01.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 2w FIFO 0,13 0t0 200047 pipe 2024-02-19T19:12:01.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 3r FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 4w FIFO 0,13 0t0 204846 pipe 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 5w REG 8,1 2615219 530450 /var/log/ceph/ceph-mgr.smithi086.huubbu.log 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 6u a_inode 0,14 0 1076 [eventpoll:7,30] 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 7r FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 8w FIFO 0,13 0t0 204848 pipe 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 9u a_inode 0,14 0 1076 [eventpoll:10,29] 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 10r FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 11w FIFO 0,13 0t0 204849 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 12u a_inode 0,14 0 1076 [eventpoll:13] 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 13r FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 14w FIFO 0,13 0t0 204850 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 15r FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 16w FIFO 0,13 0t0 204105 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 17u unix 0xffffa09a81d51e00 0t0 204106 /var/run/ceph/ceph-mgr.smithi086.huubbu.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 18r FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 19w FIFO 0,13 0t0 204110 pipe 2024-02-19T19:12:01.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 20r FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 21w FIFO 0,13 0t0 204111 pipe 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 22u IPv6 195428 0t0 TCP *:callwaveiam (LISTEN) 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 24u REG 0,82 1038 658739 /tmp/tmpkhht18rb 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 25u a_inode 0,14 0 1076 [eventpoll:22] 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 26u REG 0,82 1704 658740 /tmp/tmp3mgutg0z 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 27u IPv6 207350 0t0 TCP *:pcsync-https (LISTEN) 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 28u a_inode 0,14 0 1076 [eventpoll:27] 2024-02-19T19:12:01.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 29u IPv4 209033 0t0 TCP smithi086.front.sepia.ceph.com:46424->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr 74270 77870 dashboard ceph 30u IPv4 210887 0t0 TCP smithi086.front.sepia.ceph.com:49164->smithi039.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 77756 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 77756 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/7 77756 root txt unknown /proc/77756/exe 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 77757 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 77757 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 77757 root txt unknown /proc/77757/exe 2024-02-19T19:12:01.968 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 19520 32283 /usr/lib64/security/pam_lastlog.so 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.969 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15264 32290 /usr/lib64/security/pam_motd.so 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-02-19T19:12:01.970 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.971 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:01.972 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15232 32292 /usr/lib64/security/pam_nologin.so 2024-02-19T19:12:01.972 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 19368 32301 /usr/lib64/security/pam_sepermit.so 2024-02-19T19:12:01.972 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:01.972 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:01.972 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.972 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.973 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.974 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:01.975 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 3u unix 0xffffa09cb10bb700 0t0 204891 type=DGRAM (CONNECTED) 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 4u IPv4 204883 0t0 TCP smithi086.front.sepia.ceph.com:ssh->smithi039.front.sepia.ceph.com:55850 (ESTABLISHED) 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 6u unix 0xffffa09a887e2300 0t0 208993 type=STREAM (CONNECTED) 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 7w FIFO 0,24 0t0 6603 /run/systemd/sessions/19.ref 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77871 root 8u unix 0xffffa09a86eb9900 0t0 204895 type=STREAM (CONNECTED) 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.976 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root txt REG 8,1 968800 13744 /usr/sbin/sshd 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 357408 10120 /usr/lib64/libnss_systemd.so.2 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 19520 32283 /usr/lib64/security/pam_lastlog.so 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15184 32309 /usr/lib64/security/pam_umask.so 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 19600 10967 /usr/lib64/libpam_misc.so.0.82.1 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 19400 32305 /usr/lib64/security/pam_succeed_if.so 2024-02-19T19:12:01.977 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 514280 32422 /usr/lib64/security/pam_systemd.so 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 27536 32284 /usr/lib64/security/pam_limits.so 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15264 32290 /usr/lib64/security/pam_motd.so 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15352 32282 /usr/lib64/security/pam_keyinit.so 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 44152 32291 /usr/lib64/security/pam_namespace.so 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 27616 32299 /usr/lib64/security/pam_selinux.so 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 48392 10228 /usr/lib64/libcrack.so.2.9.0 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 32608 10961 /usr/lib64/libpwquality.so.1.0.2 2024-02-19T19:12:01.978 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15240 32287 /usr/lib64/security/pam_loginuid.so 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 56712 32310 /usr/lib64/security/pam_unix.so 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15840 32264 /usr/lib64/security/pam_pwquality.so 2024-02-19T19:12:01.979 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15232 32292 /usr/lib64/security/pam_nologin.so 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 19368 32301 /usr/lib64/security/pam_sepermit.so 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 14928 32270 /usr/lib64/security/pam_deny.so 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 15184 32274 /usr/lib64/security/pam_faildelay.so 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-02-19T19:12:01.980 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-02-19T19:12:01.981 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 4483280 10007 /usr/lib64/libcrypto.so.3.0.7 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:01.982 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 19464 32272 /usr/lib64/security/pam_env.so 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root mem REG 0,19 0 19 /sys/fs/selinux/status 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 1u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 2u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 3u unix 0xffffa09cb10bb700 0t0 204891 type=DGRAM (CONNECTED) 2024-02-19T19:12:01.983 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 4u IPv4 204883 0t0 TCP smithi086.front.sepia.ceph.com:ssh->smithi039.front.sepia.ceph.com:55850 (ESTABLISHED) 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 5u unix 0xffffa09a86eb9400 0t0 204894 type=STREAM (CONNECTED) 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 6u unix 0xffffa09a887e2300 0t0 208993 type=STREAM (CONNECTED) 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:sshd 77874 root 7w FIFO 0,24 0t0 6603 /run/systemd/sessions/19.ref 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:01.984 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:01.985 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:01.986 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 3u unix 0xffffa09a883d5000 0t0 213088 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 5u a_inode 0,14 0 1076 [eventfd:27] 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 7r FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:01.988 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 9r FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 11r FIFO 8,1 0t0 426304 /var/lib/containers/storage/overlay-containers/7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085/userdata/ctl 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 12u unix 0xffffa09bc603e900 0t0 214017 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 13w FIFO 8,1 0t0 426304 /var/lib/containers/storage/overlay-containers/7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085/userdata/ctl 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 14r FIFO 8,1 0t0 426305 /var/lib/containers/storage/overlay-containers/7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085/userdata/winsz 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 15w FIFO 8,1 0t0 426305 /var/lib/containers/storage/overlay-containers/7dc732d9913b677c323f976f866efb27a2da882ab71e88246b7848b50bd96085/userdata/winsz 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 82671 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 82673 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:01.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:01.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:01.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:01.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:01.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:01.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:01.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:01.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:01.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82676 log ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82677 service ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82678 admin_soc ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82679 signal_ha ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82680 rocksdb:l ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82681 rocksdb:h ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82697 ceph-mon ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82698 fn_monsto ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82699 msgr-work ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82700 msgr-work ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82701 msgr-work ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82702 OpHistory ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82703 fin ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82704 safe_time ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82705 cpu_tp ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82706 cpu_tp ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82707 cpu_tp ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82708 cpu_tp ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82709 ms_dispat ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82710 ms_local ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82711 safe_time ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82712 ms_dispat ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 82713 ms_local ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph cwd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph rtd DIR 0,45 4096 658742 / 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph txt REG 0,45 13891640 531020 /usr/bin/ceph-mon 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 531020 /usr/bin/ceph-mon (path inode=2803) 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 540928 /usr/lib64/ceph/libceph-common.so.2 (path inode=265449) 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 1w FIFO 0,13 0t0 210943 pipe 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 2w FIFO 0,13 0t0 210944 pipe 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 3r FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 4w FIFO 0,13 0t0 212271 pipe 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 5u IPv4 214021 0t0 TCP smithi086.front.sepia.ceph.com:3300 (LISTEN) 2024-02-19T19:12:02.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 6r FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 7w FIFO 0,13 0t0 212283 pipe 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 8u unix 0xffffa09a81d50a00 0t0 212284 /var/run/ceph/ceph-mon.smithi086.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 9r FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 10w FIFO 0,13 0t0 212285 pipe 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 11r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 12uW REG 253,4 0 33816712 /var/lib/ceph/mon/ceph-smithi086/store.db/LOCK 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 13r DIR 253,4 4096 33816711 /var/lib/ceph/mon/ceph-smithi086/store.db 2024-02-19T19:12:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 14u IPv4 237922 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37708 (ESTABLISHED) 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 15u IPv4 243956 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:57124 (ESTABLISHED) 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 16w REG 253,4 611068 33816717 /var/lib/ceph/mon/ceph-smithi086/store.db/000041.log 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 17w REG 253,4 1034 33816732 /var/lib/ceph/mon/ceph-smithi086/store.db/MANIFEST-000032 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 18u a_inode 0,14 0 1076 [eventpoll:5,15,19,28,41] 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 19r FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 20w FIFO 0,13 0t0 212294 pipe 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 21u a_inode 0,14 0 1076 [eventpoll:22,36,43] 2024-02-19T19:12:02.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 22r FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 23w FIFO 0,13 0t0 212295 pipe 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 24u a_inode 0,14 0 1076 [eventpoll:14,25,29,42] 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 25r FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 26w FIFO 0,13 0t0 212296 pipe 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 27w REG 8,1 23558822 530547 /var/log/ceph/ceph-mon.smithi086.log 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 28u IPv4 214022 0t0 TCP smithi086.front.sepia.ceph.com:smc-https (LISTEN) 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 29u IPv4 214023 0t0 TCP smithi086.front.sepia.ceph.com:37580->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 30r FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 31w FIFO 0,13 0t0 205759 pipe 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 32r FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 33w FIFO 0,13 0t0 205760 pipe 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 34r FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 35w FIFO 0,13 0t0 205761 pipe 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 36u IPv4 209629 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi039.front.sepia.ceph.com:40284 (ESTABLISHED) 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 37u unix 0xffffa09bc603c600 0t0 214026 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 38w REG 8,1 119310 530575 /var/log/ceph/ceph.log 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 39w REG 8,1 255621 530574 /var/log/ceph/ceph.audit.log 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 40w REG 8,1 26554 528405 /var/log/ceph/ceph.cephadm.log 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 41u IPv4 224232 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:45204 (ESTABLISHED) 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 42u IPv4 253231 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:39824 (ESTABLISHED) 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 43u IPv4 246537 0t0 TCP smithi086.front.sepia.ceph.com:3300->smithi086.front.sepia.ceph.com:37822 (ESTABLISHED) 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon 82675 88678 rstore_co ceph 44r REG 253,4 8399355 33816723 /var/lib/ceph/mon/ceph-smithi086/store.db/000043.sst 2024-02-19T19:12:02.282 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 87320 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 87320 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/u 87320 root txt unknown /proc/87320/exe 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 87440 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 87440 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/4 87440 root txt unknown /proc/87440/exe 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.283 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:02.284 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:02.285 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:02.286 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 3u unix 0xffffa09cb10bf800 0t0 215962 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 5u a_inode 0,14 0 1076 [eventfd:22] 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.287 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 7r FIFO 0,13 0t0 217820 pipe 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 9r FIFO 0,13 0t0 217821 pipe 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 10u unix 0xffffa09a887c1400 0t0 217829 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 11r FIFO 8,1 0t0 426312 /var/lib/containers/storage/overlay-containers/1c6a2621732bde6e4513cab794c44b3bdb138cabcfa30bf9d9181e091bfa55bf/userdata/ctl 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 12u unix 0xffffa09a887c2800 0t0 217822 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 13w FIFO 8,1 0t0 426312 /var/lib/containers/storage/overlay-containers/1c6a2621732bde6e4513cab794c44b3bdb138cabcfa30bf9d9181e091bfa55bf/userdata/ctl 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 14r FIFO 8,1 0t0 426313 /var/lib/containers/storage/overlay-containers/1c6a2621732bde6e4513cab794c44b3bdb138cabcfa30bf9d9181e091bfa55bf/userdata/winsz 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 15w FIFO 8,1 0t0 426313 /var/lib/containers/storage/overlay-containers/1c6a2621732bde6e4513cab794c44b3bdb138cabcfa30bf9d9181e091bfa55bf/userdata/winsz 2024-02-19T19:12:02.288 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 87961 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root cwd DIR 0,47 4096 658772 / 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root rtd DIR 0,47 4096 658772 / 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root 1w FIFO 0,13 0t0 217820 pipe 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root 2w FIFO 0,13 0t0 217821 pipe 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 87963 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph cwd DIR 0,47 4096 658772 / 2024-02-19T19:12:02.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph rtd DIR 0,47 4096 658772 / 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph txt REG 0,47 11864 407097 /usr/libexec/platform-python3.6 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 407097 /usr/libexec/platform-python3.6 (stat: No such file or directory) 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406020 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406042 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406034 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406023 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406033 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406026 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 406011 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (stat: No such file or directory) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 404233 /usr/lib64/libutil-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 404159 /usr/lib64/libpython3.6m.so.1.0 (stat: No such file or directory) 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph 0u CHR 1,3 0t0 5 /dev/null 2024-02-19T19:12:02.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph 1w FIFO 0,13 0t0 217820 pipe 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-cras 87965 ceph 2w FIFO 0,13 0t0 217821 pipe 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 91604 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 91604 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/3 91604 root txt unknown /proc/91604/exe 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 91649 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 91649 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 91649 root txt unknown /proc/91649/exe 2024-02-19T19:12:02.293 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 92327 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 92327 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/2 92327 root txt unknown /proc/92327/exe 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:02.294 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:02.295 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:02.296 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:02.297 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 3u unix 0xffffa09a818ad500 0t0 225920 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 5u a_inode 0,14 0 1076 [eventfd:19] 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 7r FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.298 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 9r FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 10u unix 0xffffa09cb10be900 0t0 224001 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 11r FIFO 8,1 0t0 426320 /var/lib/containers/storage/overlay-containers/37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f/userdata/ctl 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 12u unix 0xffffa09bc603df00 0t0 229868 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 13w FIFO 8,1 0t0 426320 /var/lib/containers/storage/overlay-containers/37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f/userdata/ctl 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 14r FIFO 8,1 0t0 426321 /var/lib/containers/storage/overlay-containers/37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f/userdata/winsz 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 15w FIFO 8,1 0t0 426321 /var/lib/containers/storage/overlay-containers/37ce649da9a62e5396c45a2a1cea26837c2bd11f3b23779cda901ff9cfab855f/userdata/winsz 2024-02-19T19:12:02.299 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 92644 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.300 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 92646 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92649 log ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92651 msgr-work ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92652 msgr-work ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92653 msgr-work ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92657 service ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92658 admin_soc ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92736 signal_ha ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92737 io_contex ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92738 io_contex ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92831 OpHistory ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 92832 ceph_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93034 ceph-osd ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93035 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93036 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93037 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93039 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93040 fn_anonym ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93045 rocksdb:l ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93046 rocksdb:l ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93047 rocksdb:l ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93048 rocksdb:h ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93606 bstore_ai ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93607 bstore_di ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93873 bstore_ai ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 93874 bstore_di ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94056 ceph-osd ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94057 cfin ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94058 bstore_kv ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94059 bstore_kv ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94060 bstore_me ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94062 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94063 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:02.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:02.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:02.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:02.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:02.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94064 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:02.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:02.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:02.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:02.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:02.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:02.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:02.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:02.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94065 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94066 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94067 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94068 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.068 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:01 smithi086 ceph-mon[82675]: pgmap v212: 65 pgs: 65 active+clean; 171 MiB data, 815 MiB used, 536 GiB / 536 GiB avail; 4.0 MiB/s rd, 5.5 MiB/s wr, 422 op/s 2024-02-19T19:12:03.068 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:01 smithi086 ceph-mon[82675]: Upgrade: Updating ceph-exporter.smithi039 (1/2) 2024-02-19T19:12:03.068 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:01 smithi086 ceph-mon[82675]: Deploying daemon ceph-exporter.smithi039 on smithi039 2024-02-19T19:12:03.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94069 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94070 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94071 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94072 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94073 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94074 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94075 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94076 ms_dispat ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94077 ms_local ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94078 fn_anonym ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94079 finisher ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94080 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94081 safe_time ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94082 ceph-osd ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94083 osd_srv_a ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94084 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94085 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94086 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94087 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94088 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94089 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94090 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94091 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94092 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94093 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.622 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.624 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.625 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.626 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.627 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.630 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.631 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.632 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94094 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94095 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94096 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94097 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94098 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94099 tp_osd_tp ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph cwd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph rtd DIR 0,96 4096 658793 / 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph txt REG 0,96 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph DEL REG 0,17 227308 /[aio] 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph DEL REG 0,17 230832 /[aio] 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 1w FIFO 0,13 0t0 229866 pipe 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 2w FIFO 0,13 0t0 229867 pipe 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 3r FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 4w FIFO 0,13 0t0 227720 pipe 2024-02-19T19:12:03.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 5w REG 8,1 197112253 530618 /var/log/ceph/ceph-osd.3.log 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,68,69,71,73,77,81,82,87,90] 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 7r FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 8w FIFO 0,13 0t0 227732 pipe 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 9u a_inode 0,14 0 1076 [eventpoll:10,57,64,74,76,78,79,80,83,85,88,91] 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 10r FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 11w FIFO 0,13 0t0 227733 pipe 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 12u a_inode 0,14 0 1076 [eventpoll:13,62,65,66,67,70,72,75,84,86] 2024-02-19T19:12:03.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 13r FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 14w FIFO 0,13 0t0 227734 pipe 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 15r FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 16w FIFO 0,13 0t0 228710 pipe 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 17u unix 0xffffa09cb1847300 0t0 228711 /var/run/ceph/ceph-osd.3.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 18u IPv4 225959 0t0 TCP *:6800 (LISTEN) 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 19u IPv4 225961 0t0 TCP *:acnet (LISTEN) 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 20u IPv4 225964 0t0 TCP *:6802 (LISTEN) 2024-02-19T19:12:03.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 21u IPv4 225968 0t0 TCP *:6803 (LISTEN) 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 22u IPv4 225973 0t0 TCP *:6804 (LISTEN) 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 23u IPv4 225979 0t0 TCP *:6805 (LISTEN) 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 24u IPv4 228725 0t0 TCP *:6806 (LISTEN) 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 25u IPv4 228733 0t0 TCP *:6807 (LISTEN) 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 26r FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 27w FIFO 0,13 0t0 230611 pipe 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 28r FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 29w FIFO 0,13 0t0 230612 pipe 2024-02-19T19:12:03.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 30u sock 0,8 0t0 226107 protocol: UDPv6 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 31r DIR 253,4 4096 50331789 /var/lib/ceph/osd/ceph-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 32uW REG 253,4 37 50331803 /var/lib/ceph/osd/ceph-3/fsid 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 33u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 34u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 35u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 36u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 37u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 38u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 39u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 40u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 41u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 42u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 43u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 44u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 45u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 46u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 47u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 48u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 49u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 50u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 51u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 52u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 53u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 54u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 55u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 56u BLK 253,3 0t0 715 /dev/dm-3 2024-02-19T19:12:03.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 57u IPv4 227309 0t0 TCP smithi086.front.sepia.ceph.com:45204->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 58r FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 59w FIFO 0,13 0t0 230058 pipe 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 60r FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 61w FIFO 0,13 0t0 230059 pipe 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 62u IPv4 230840 0t0 TCP smithi086.front.sepia.ceph.com:37772->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 63u IPv4 228996 0t0 TCP smithi086.front.sepia.ceph.com:55944->smithi039.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 64u IPv4 230907 0t0 TCP smithi086.front.sepia.ceph.com:34916->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 65u IPv4 237010 0t0 TCP smithi086.front.sepia.ceph.com:42036->smithi086.front.sepia.ceph.com:6810 (ESTABLISHED) 2024-02-19T19:12:03.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 66u IPv4 231725 0t0 TCP smithi086.front.sepia.ceph.com:42752->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 67u IPv4 247876 0t0 TCP smithi086.front.sepia.ceph.com:49066->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 68u IPv4 231727 0t0 TCP smithi086.front.sepia.ceph.com:36640->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 69u IPv4 231728 0t0 TCP smithi086.front.sepia.ceph.com:33410->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 70u IPv4 239982 0t0 TCP smithi086.front.sepia.ceph.com:52050->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 71u IPv4 231733 0t0 TCP smithi086.front.sepia.ceph.com:55212->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 72u IPv4 231734 0t0 TCP smithi086.front.sepia.ceph.com:54216->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 73u IPv4 232505 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42684 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 74u IPv4 228997 0t0 TCP smithi086.front.sepia.ceph.com:47322->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 75u IPv4 247877 0t0 TCP smithi086.front.sepia.ceph.com:41140->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 76u IPv4 236046 0t0 TCP smithi086.front.sepia.ceph.com:41490->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 77u IPv4 237999 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:52382 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 78u IPv4 229003 0t0 TCP smithi086.front.sepia.ceph.com:52996->smithi039.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 79u IPv4 246986 0t0 TCP smithi086.front.sepia.ceph.com:57096->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 80u IPv4 237011 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:42010 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 81u IPv4 232499 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42664 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 82u IPv4 243082 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi086.front.sepia.ceph.com:34906 (ESTABLISHED) 2024-02-19T19:12:03.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 83u IPv4 246070 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi086.front.sepia.ceph.com:57350 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 84u IPv4 231753 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46796 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 85u IPv4 232502 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46812 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 86u IPv4 231756 0t0 TCP smithi086.front.sepia.ceph.com:6804->smithi039.front.sepia.ceph.com:46828 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 87u IPv4 231759 0t0 TCP smithi086.front.sepia.ceph.com:6806->smithi039.front.sepia.ceph.com:42674 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 88u IPv4 255406 0t0 TCP smithi086.front.sepia.ceph.com:acnet->192.168.0.1:48904 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 90u IPv4 252340 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:50158 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 92648 94100 osd_srv_h ceph 91u IPv4 250653 0t0 TCP smithi086.front.sepia.ceph.com:6800->smithi039.front.sepia.ceph.com:44348 (ESTABLISHED) 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 93589 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:03.764 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 93589 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:kworker/1 93589 root txt unknown /proc/93589/exe 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:03.765 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:03.766 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:03.767 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 3u unix 0xffffa09a8c991900 0t0 233954 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:03.768 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 5u a_inode 0,14 0 1076 [eventfd:23] 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 7r FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 9r FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 10u unix 0xffffa09a83ad8a00 0t0 237753 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 11r FIFO 8,1 0t0 658836 /var/lib/containers/storage/overlay-containers/f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee/userdata/ctl 2024-02-19T19:12:03.769 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 12u unix 0xffffa09a88eb8000 0t0 235692 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 13w FIFO 8,1 0t0 658836 /var/lib/containers/storage/overlay-containers/f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee/userdata/ctl 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 14r FIFO 8,1 0t0 658837 /var/lib/containers/storage/overlay-containers/f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee/userdata/winsz 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 15w FIFO 8,1 0t0 658837 /var/lib/containers/storage/overlay-containers/f6b44d6c56416325e7239a1a8b429ea91a9e9d4c668f8230fb35bfd53731cfee/userdata/winsz 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 97977 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 97979 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.797 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.798 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97982 log ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97984 msgr-work ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97985 msgr-work ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97986 msgr-work ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97990 service ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 97991 admin_soc ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98069 signal_ha ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98070 io_contex ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:03.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98071 io_contex ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:03.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:03.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:03.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:03.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:03.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:03.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:03.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:03.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:03.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:03.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:03.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:03.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:03.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:03.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:03.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:03.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:03.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:03.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.001 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.002 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98162 OpHistory ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98163 ceph_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98396 ceph-osd ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98397 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98398 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98399 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.108 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.109 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.110 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.111 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98400 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98401 fn_anonym ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98407 rocksdb:l ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.182 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.183 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.184 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.186 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98408 rocksdb:l ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98409 rocksdb:l ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.209 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.210 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98410 rocksdb:h ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98937 bstore_ai ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 98938 bstore_di ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.277 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.278 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.279 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.280 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.281 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.282 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.283 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.284 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.285 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99199 bstore_ai ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99200 bstore_di ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99382 ceph-osd ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99383 cfin ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99384 bstore_kv ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99385 bstore_kv ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99386 bstore_me ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99388 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99389 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99390 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99391 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99392 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99393 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99394 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99395 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99396 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99397 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.621 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:12:03 smithi039 ceph-mon[92224]: pgmap v213: 65 pgs: 65 active+clean; 171 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 421 op/s 2024-02-19T19:12:04.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.633 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.634 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.635 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.636 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.637 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99398 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.638 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.639 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.641 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99399 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.659 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.660 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.661 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.662 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.663 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.664 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.665 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.667 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.668 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.671 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.672 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.673 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.674 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.675 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99400 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.676 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.678 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.680 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.681 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.682 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.683 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.684 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.685 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.686 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.687 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.688 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.689 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.690 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.691 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.692 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.693 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.694 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99401 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.696 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.697 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.698 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.699 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.700 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.701 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.702 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.703 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.704 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.706 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.707 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.708 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.711 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99402 ms_dispat ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.713 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.714 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.715 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.716 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.717 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.719 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.720 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.722 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.723 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.724 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.725 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.727 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.728 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.729 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.730 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.731 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99403 ms_local ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.732 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.733 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.734 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.735 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.736 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.738 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.739 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.741 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.742 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.743 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.744 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.745 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.746 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.747 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.748 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.749 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99404 fn_anonym ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.750 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.751 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.752 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.753 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.754 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.755 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.756 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.757 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.758 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.759 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.760 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.761 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.762 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.764 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.765 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.766 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.767 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.768 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.769 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99405 finisher ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.770 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.771 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.772 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.774 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.775 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.776 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.777 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.778 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.779 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.780 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.781 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.782 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.783 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.784 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.785 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.786 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.787 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.788 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99406 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.789 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.791 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.792 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.793 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.794 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.795 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.799 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.802 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.803 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.805 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.808 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.809 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.810 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.811 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99407 safe_time ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.812 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.813 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.814 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.816 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.817 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.819 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.820 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.821 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.823 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.824 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.825 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.826 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.827 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.829 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.830 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99408 ceph-osd ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.831 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.833 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.834 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.835 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.837 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.838 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.839 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.840 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.842 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.843 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.845 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.846 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.847 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99409 osd_srv_a ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.850 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.851 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.852 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.853 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.854 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.855 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.856 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.857 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.859 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.860 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.861 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.862 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.864 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.865 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.867 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99410 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.869 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.870 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.872 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.874 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.875 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.876 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.877 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.878 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.879 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.880 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.881 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.884 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.885 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.886 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.887 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99411 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.888 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.889 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.890 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.891 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.892 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.893 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.895 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.896 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.897 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.898 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.899 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.900 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.901 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.903 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.904 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.905 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99412 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.906 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.907 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.908 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.909 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.910 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.911 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.912 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.913 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.914 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.915 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.916 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.918 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.919 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.920 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.922 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.923 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.924 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99413 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.926 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.927 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.929 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.930 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.931 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.936 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.937 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.939 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.940 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.941 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.942 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.943 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99414 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.944 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.945 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.946 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.947 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.948 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.949 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.950 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.951 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.953 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.954 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.955 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.958 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.960 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.961 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99415 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.962 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.963 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.966 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.967 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.968 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.969 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.970 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.971 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.973 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.974 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.975 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.976 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.977 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.978 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.979 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.980 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99416 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:04.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:04.983 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:04.984 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:04.986 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:04.987 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:04.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:04.989 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:04.990 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:04.991 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:04.992 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.993 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.994 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:04.995 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:04.996 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:04.998 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99417 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:04.999 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.000 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.004 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.005 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.006 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.007 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.008 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.009 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.010 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.011 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.012 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.013 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.014 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.015 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.016 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.017 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.018 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.019 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99418 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.020 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.021 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.022 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.023 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.024 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.026 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.027 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.028 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.029 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.031 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.032 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.033 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.034 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.035 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.036 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.037 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.038 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99419 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.039 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.040 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.042 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.043 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.044 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.045 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.046 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.048 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.049 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.050 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.051 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.052 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.053 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.054 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.055 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.056 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.057 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99420 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.058 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.059 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.061 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.062 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.063 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.064 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.065 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.066 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.067 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.069 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.071 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.072 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.073 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.074 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.075 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.076 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.077 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99421 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.079 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.080 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.081 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.082 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.083 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.086 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.087 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.088 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.089 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.090 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.091 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.092 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.093 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.094 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.095 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.096 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.097 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99422 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.099 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.100 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.101 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.103 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.104 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.105 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.106 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.107 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.107 INFO:journalctl@ceph.mon.smithi086.smithi086.stdout:Feb 19 19:12:03 smithi086 ceph-mon[82675]: pgmap v213: 65 pgs: 65 active+clean; 171 MiB data, 825 MiB used, 536 GiB / 536 GiB avail; 5.3 MiB/s rd, 5.1 MiB/s wr, 421 op/s 2024-02-19T19:12:05.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.112 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.113 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.117 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.118 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.119 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.120 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.121 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99423 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.122 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.123 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.124 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.125 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.126 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.127 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.128 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.129 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.130 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.132 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.133 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.134 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.135 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.136 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.138 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.140 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.141 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99424 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.142 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.143 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.144 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.145 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.146 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.147 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.148 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.151 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.153 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.154 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.155 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.156 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.157 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.158 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.159 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.160 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99425 tp_osd_tp ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph cwd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.161 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph rtd DIR 0,100 4096 658821 / 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph txt REG 0,100 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.162 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.163 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.164 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.165 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.166 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.167 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.168 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.169 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph DEL REG 0,17 233302 /[aio] 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph DEL REG 0,17 233287 /[aio] 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 1w FIFO 0,13 0t0 235690 pipe 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 2w FIFO 0,13 0t0 235691 pipe 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 3r FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 4w FIFO 0,13 0t0 234850 pipe 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 5w REG 8,1 142746914 530656 /var/log/ceph/ceph-osd.4.log 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,65,66,69,74,75,82,83,84,87,90] 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 7r FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 8w FIFO 0,13 0t0 234862 pipe 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,72,78,79,81,85,88,91] 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 10r FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 11w FIFO 0,13 0t0 234863 pipe 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,63,68,71,73,76,77,80,86] 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 13r FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 14w FIFO 0,13 0t0 234864 pipe 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 15r FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 16w FIFO 0,13 0t0 233962 pipe 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 17u unix 0xffffa09bb0ff8500 0t0 233963 /var/run/ceph/ceph-osd.4.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 18u IPv4 236727 0t0 TCP *:6808 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 19u IPv4 236737 0t0 TCP *:6809 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 20u IPv4 236748 0t0 TCP *:6810 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 21u IPv4 236760 0t0 TCP *:6811 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 22u IPv4 236773 0t0 TCP *:6812 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 23u IPv4 236787 0t0 TCP *:6813 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 24u IPv4 236802 0t0 TCP *:6814 (LISTEN) 2024-02-19T19:12:05.173 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 25u IPv4 236818 0t0 TCP *:6815 (LISTEN) 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 26r FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 27w FIFO 0,13 0t0 233002 pipe 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 28r FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 29w FIFO 0,13 0t0 233003 pipe 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 30u sock 0,8 0t0 233157 protocol: UDPv6 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 31r DIR 253,4 4096 150 /var/lib/ceph/osd/ceph-4 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 32uW REG 253,4 37 171 /var/lib/ceph/osd/ceph-4/fsid 2024-02-19T19:12:05.174 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 33u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 34u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 35u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 36u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 37u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 38u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 39u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 40u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 41u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 42u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 43u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 44u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 45u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 46u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 47u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 48u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 49u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 50u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 51u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 52u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 53u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 54u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 55u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 56u BLK 253,2 0t0 696 /dev/dm-2 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 57r FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.177 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 58u IPv4 233305 0t0 TCP smithi086.front.sepia.ceph.com:37708->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 59w FIFO 0,13 0t0 237965 pipe 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 60r FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 61w FIFO 0,13 0t0 237966 pipe 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 62u IPv4 237967 0t0 TCP smithi086.front.sepia.ceph.com:47320->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 63u IPv4 241413 0t0 TCP smithi086.front.sepia.ceph.com:41154->smithi086.front.sepia.ceph.com:6818 (ESTABLISHED) 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 64u IPv4 246990 0t0 TCP smithi086.front.sepia.ceph.com:57104->smithi086.front.sepia.ceph.com:6820 (ESTABLISHED) 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 65u IPv4 242074 0t0 TCP smithi086.front.sepia.ceph.com:49070->smithi086.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 66u IPv4 237982 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi086.front.sepia.ceph.com:42036 (ESTABLISHED) 2024-02-19T19:12:05.178 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 67u IPv4 237983 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40874 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 68u IPv4 237986 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:52050 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 69u IPv4 237987 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52212 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 70u IPv4 237990 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:41490 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 71u IPv4 237993 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39576 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 72u IPv4 239983 0t0 TCP smithi086.front.sepia.ceph.com:42622->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 73u IPv4 245123 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi086.front.sepia.ceph.com:55190 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 74u IPv4 239984 0t0 TCP smithi086.front.sepia.ceph.com:49818->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 75u IPv4 241441 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi086.front.sepia.ceph.com:34976 (ESTABLISHED) 2024-02-19T19:12:05.179 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 76u IPv4 237996 0t0 TCP smithi086.front.sepia.ceph.com:51428->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 77u IPv4 237997 0t0 TCP smithi086.front.sepia.ceph.com:36742->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 78u IPv4 238002 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52226 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 79u IPv4 239986 0t0 TCP smithi086.front.sepia.ceph.com:39668->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 80u IPv4 238005 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39602 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 81u IPv4 239987 0t0 TCP smithi086.front.sepia.ceph.com:52382->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 82u IPv4 239988 0t0 TCP smithi086.front.sepia.ceph.com:38024->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 83u IPv4 239989 0t0 TCP smithi086.front.sepia.ceph.com:42010->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 84u IPv4 237014 0t0 TCP smithi086.front.sepia.ceph.com:6814->smithi039.front.sepia.ceph.com:39592 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 85u IPv4 237017 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40890 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 86u IPv4 237020 0t0 TCP smithi086.front.sepia.ceph.com:6812->smithi039.front.sepia.ceph.com:52216 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 87u IPv4 237023 0t0 TCP smithi086.front.sepia.ceph.com:6810->smithi039.front.sepia.ceph.com:40898 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 88u IPv4 254413 0t0 TCP smithi086.front.sepia.ceph.com:6809->192.168.0.1:57062 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 90u IPv4 256027 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:51020 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 97981 99426 osd_srv_h ceph 91u IPv4 253581 0t0 TCP smithi086.front.sepia.ceph.com:6808->smithi039.front.sepia.ceph.com:60944 (ESTABLISHED) 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root cwd DIR 8,1 4096 2 / 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root rtd DIR 8,1 4096 2 / 2024-02-19T19:12:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root txt REG 8,1 158208 15302 /usr/bin/conmon 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-02-19T19:12:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-02-19T19:12:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 489696 10000 /usr/lib64/libpcre.so.1.2.12 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 915808 10122 /usr/lib64/libsystemd.so.0.35.0 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 1303016 10167 /usr/lib64/libglib-2.0.so.0.6800.4 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-02-19T19:12:05.184 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 0r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 1w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 2w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 3u unix 0xffffa09a818a9e00 0t0 235466 type=SEQPACKET (CONNECTED) 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 4r CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 5u a_inode 0,14 0 1076 [eventfd:24] 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 6w CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.185 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 7r FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 8r a_inode 0,14 0 1076 inotify 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 9r FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 10u unix 0xffffa09a83adf300 0t0 242759 type=DGRAM (UNCONNECTED) 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 11r FIFO 8,1 0t0 658872 /var/lib/containers/storage/overlay-containers/5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400/userdata/ctl 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 12u unix 0xffffa09a883d0500 0t0 239573 /proc/self/fd/11/attach type=SEQPACKET (LISTEN) 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 13w FIFO 8,1 0t0 658872 /var/lib/containers/storage/overlay-containers/5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400/userdata/ctl 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 14r FIFO 8,1 0t0 658873 /var/lib/containers/storage/overlay-containers/5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400/userdata/winsz 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 15w FIFO 8,1 0t0 658873 /var/lib/containers/storage/overlay-containers/5b366510b403b6c60312da85d4f98692f35111c93013f9105ec53424ba200400/userdata/winsz 2024-02-19T19:12:05.186 INFO:teuthology.orchestra.run.smithi086.stdout:conmon 103314 root 16u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root txt REG 8,1 810096 1960 /run/podman-init 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:podman-in 103316 root 3u a_inode 0,14 0 1076 [signalfd] 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.187 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.188 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.189 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.190 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.192 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.194 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.195 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.196 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.197 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.198 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.199 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.200 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.201 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.203 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.204 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.206 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.207 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.208 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.211 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.212 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.213 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.214 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.215 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.216 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.217 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.218 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.219 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.221 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.222 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.223 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.226 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.227 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.228 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.229 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.230 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103319 log ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.231 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.232 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.233 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.235 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.236 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.237 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.238 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.240 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.241 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.242 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.243 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.244 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.245 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.246 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.247 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.248 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.249 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103321 msgr-work ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.250 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.251 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.252 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.253 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.254 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.255 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.256 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.257 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.258 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.259 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.260 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.261 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.262 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.264 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.265 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.266 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.268 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103322 msgr-work ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.269 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.270 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.271 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.273 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.274 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.275 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.276 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.277 INFO:tasks.workunit.client.1.smithi086.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-19T19:12:05.277 INFO:tasks.workunit.client.1.smithi086.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-19T19:12:05.277 INFO:tasks.workunit.client.1.smithi086.stderr:+ make 2024-02-19T19:12:05.278 INFO:tasks.workunit.client.1.smithi086.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-19T19:12:05.286 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.287 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.288 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.290 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.291 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.292 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.293 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.294 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.295 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.296 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.297 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103323 msgr-work ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.298 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.299 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.300 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.301 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.302 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.303 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.304 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.305 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.306 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.307 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.308 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.309 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.310 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.311 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.312 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.313 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.314 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.315 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.316 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.317 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.318 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.319 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103327 service ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.322 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.323 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.324 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.327 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.328 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.329 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.331 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.332 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.334 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.335 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.336 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.337 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.338 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.340 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103328 admin_soc ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.341 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.343 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.344 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.345 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.346 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.347 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.348 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.349 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.350 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.351 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.353 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.354 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.355 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.357 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.358 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.359 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.361 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103406 signal_ha ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.362 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.363 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.364 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.365 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.367 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.368 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.373 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.374 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.375 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.377 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.378 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.379 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.380 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103407 io_contex ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.381 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.386 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.387 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.388 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.390 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.391 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.392 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.393 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.395 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.396 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.397 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.398 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.399 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103408 io_contex ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.401 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.402 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.403 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.404 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.406 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.407 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.408 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.409 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.411 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.412 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.413 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.414 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.415 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.416 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.417 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.418 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.419 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.420 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103499 OpHistory ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.421 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.423 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.424 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.425 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.426 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.427 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.428 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.429 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.430 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.431 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.432 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.433 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.434 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.435 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.436 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.437 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.438 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.439 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.441 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103500 ceph_time ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.442 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.443 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.445 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.446 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.447 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.448 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.449 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.450 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.451 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.452 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.453 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.454 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.455 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.457 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.458 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.460 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.461 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103708 ceph-osd ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.462 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.463 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.465 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.466 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.467 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.468 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.469 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.470 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.471 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.472 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.473 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.474 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.476 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.477 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.478 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.479 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.480 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.481 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.482 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103709 safe_time ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.490 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.491 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.492 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.493 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.494 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.495 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.496 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.498 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.499 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.500 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.501 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103710 safe_time ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.502 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.503 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.505 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.507 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.508 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.512 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.513 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.514 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.515 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.516 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.517 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.519 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.520 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103711 safe_time ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.522 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.524 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.525 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.526 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.527 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.528 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.529 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.531 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.532 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.533 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.535 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.536 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.537 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.538 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.540 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103712 safe_time ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.541 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.542 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.545 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.546 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.547 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.548 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.549 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.550 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.552 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.553 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.554 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.555 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.556 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.557 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.558 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.559 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.560 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103713 fn_anonym ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.561 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.563 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.564 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.565 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.566 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.567 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.568 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.570 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.571 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.573 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.574 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.575 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.576 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.577 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.578 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.580 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103720 rocksdb:l ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.582 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.583 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.584 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.586 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.587 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.588 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.589 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.590 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.591 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.592 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.593 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.594 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.595 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.596 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.597 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:05.598 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 70u IPv4 241420 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54268 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 71u IPv4 241421 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49070 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 72u IPv4 241424 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57104 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 73u IPv4 241427 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50166 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 74u IPv4 241430 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34556 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 75u IPv4 243079 0t0 TCP smithi086.front.sepia.ceph.com:49746->smithi039.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 76u IPv4 243080 0t0 TCP smithi086.front.sepia.ceph.com:41150->smithi039.front.sepia.ceph.com:6824 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 77u IPv4 243081 0t0 TCP smithi086.front.sepia.ceph.com:34906->smithi086.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.599 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 78u IPv4 241433 0t0 TCP smithi086.front.sepia.ceph.com:32952->smithi039.front.sepia.ceph.com:6804 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 79u IPv4 241434 0t0 TCP smithi086.front.sepia.ceph.com:46138->smithi039.front.sepia.ceph.com:6816 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 80u IPv4 241435 0t0 TCP smithi086.front.sepia.ceph.com:46974->smithi039.front.sepia.ceph.com:6822 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 81u IPv4 241436 0t0 TCP smithi086.front.sepia.ceph.com:55190->smithi086.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 82u IPv4 241437 0t0 TCP smithi086.front.sepia.ceph.com:41022->smithi039.front.sepia.ceph.com:6808 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 83u IPv4 241438 0t0 TCP smithi086.front.sepia.ceph.com:40272->smithi039.front.sepia.ceph.com:6814 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 84u IPv4 241439 0t0 TCP smithi086.front.sepia.ceph.com:57350->smithi086.front.sepia.ceph.com:6806 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 85u IPv4 241440 0t0 TCP smithi086.front.sepia.ceph.com:34976->smithi086.front.sepia.ceph.com:6812 (ESTABLISHED) 2024-02-19T19:12:05.600 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 87u IPv4 241447 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54272 (ESTABLISHED) 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 88u IPv4 241450 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50182 (ESTABLISHED) 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 89u IPv4 252692 0t0 TCP smithi086.front.sepia.ceph.com:pentbox-sim->192.168.0.1:53324 (ESTABLISHED) 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 90u IPv4 252343 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:37026 (ESTABLISHED) 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103721 rocksdb:l ceph 91u IPv4 254650 0t0 TCP smithi086.front.sepia.ceph.com:6816->smithi039.front.sepia.ceph.com:43078 (ESTABLISHED) 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph cwd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph rtd DIR 0,94 4096 658857 / 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph txt REG 0,94 31927808 531023 /usr/bin/ceph-osd 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 531023 /usr/bin/ceph-osd (path inode=2790) 2024-02-19T19:12:05.601 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546062 /usr/lib64/rados-classes/libcls_lock.so.1.0.0 (path inode=265672) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546089 /usr/lib64/rados-classes/libcls_rgw_gc.so.1.0.0 (path inode=265700) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404075 /usr/lib64/liblua-5.3.so (stat: No such file or directory) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546068 /usr/lib64/rados-classes/libcls_lua.so.1.0.0 (path inode=265678) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546041 /usr/lib64/rados-classes/libcls_2pc_queue.so.1.0.0 (path inode=265651) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546077 /usr/lib64/rados-classes/libcls_queue.so.1.0.0 (path inode=265688) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546059 /usr/lib64/rados-classes/libcls_journal.so.1.0.0 (path inode=265669) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546086 /usr/lib64/rados-classes/libcls_rgw.so.1.0.0 (path inode=265697) 2024-02-19T19:12:05.602 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546053 /usr/lib64/rados-classes/libcls_fifo.so.1.0.0 (path inode=265663) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546083 /usr/lib64/rados-classes/libcls_refcount.so.1.0.0 (path inode=265694) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541488 /usr/lib64/liboath.so.0.1.3 (path inode=2299) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 540921 /usr/lib64/ceph/erasure-code/libec_lrc.so (path inode=265641) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404121 /usr/lib64/libnss_files-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541484 /usr/lib64/libnss_sss.so.2 (path inode=11389) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541606 /usr/lib64/liburcu-common.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541604 /usr/lib64/liburcu-cds.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541602 /usr/lib64/liburcu-bp.so.6.0.0 (stat: No such file or directory) 2024-02-19T19:12:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541441 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 (path inode=2261) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404144 /usr/lib64/libpcre2-8.so.0.7.1 (stat: No such file or directory) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404174 /usr/lib64/librt-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404180 /usr/lib64/libselinux.so.1 (path inode=9824) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404058 /usr/lib64/libkeyutils.so.1.6 (stat: No such file or directory) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404066 /usr/lib64/libkrb5support.so.0.1 (path inode=11405) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541473 /usr/lib64/libnl-3.so.200.26.0 (path inode=9812) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541481 /usr/lib64/libnl-route-3.so.200.26.0 (path inode=9820) 2024-02-19T19:12:05.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404096 /usr/lib64/libmount.so.1.1.0 (path inode=10156) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403974 /usr/lib64/libcom_err.so.2.1 (path inode=9711) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404052 /usr/lib64/libk5crypto.so.3.1 (path inode=11397) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404064 /usr/lib64/libkrb5.so.3.3 (path inode=11403) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404036 /usr/lib64/libgssapi_krb5.so.2.2 (path inode=11393) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404196 /usr/lib64/libssl.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404110 /usr/lib64/libnghttp2.so.14.17.0 (stat: No such file or directory) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404236 /usr/lib64/libuuid.so.1.3.0 (path inode=9712) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541600 /usr/lib64/libunwind.so.8.0.1 (path inode=2325) 2024-02-19T19:12:05.605 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403967 /usr/lib64/libc-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404011 /usr/lib64/libgcc_s-8-20210514.so.1 (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404082 /usr/lib64/libm-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404198 /usr/lib64/libstdc++.so.6.0.25 (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541531 /usr/lib64/librdmacm.so.1.3.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541383 /usr/lib64/libibverbs.so.1.14.48.0 (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404226 /usr/lib64/libudev.so.1.6.11 (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404155 /usr/lib64/libpthread-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.606 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403981 /usr/lib64/libcrypto.so.1.1.1k (stat: No such file or directory) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403991 /usr/lib64/libdl-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404164 /usr/lib64/libresolv-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403972 /usr/lib64/libcap.so.2.48 (path inode=9702) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541590 /usr/lib64/libthrift-0.13.0.so (stat: No such file or directory) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403985 /usr/lib64/libcurl.so.4.5.0 (stat: No such file or directory) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404244 /usr/lib64/libz.so.1.2.11 (path inode=9686) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404077 /usr/lib64/liblz4.so.1.8.3 (stat: No such file or directory) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 404190 /usr/lib64/libsnappy.so.1.1.8 (path inode=10813) 2024-02-19T19:12:05.607 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541284 /usr/lib64/libaio.so.1.0.1 (path inode=9879) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541359 /usr/lib64/libfuse.so.2.9.7 (stat: No such file or directory) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403962 /usr/lib64/libblkid.so.1.1.0 (path inode=10126) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 541572 /usr/lib64/libtcmalloc.so.4.5.3 (stat: No such file or directory) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 403938 /usr/lib64/ld-2.28.so (stat: No such file or directory) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546095 /usr/lib64/rados-classes/libcls_test_remote_reads.so.1.0.0 (path inode=265706) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546065 /usr/lib64/rados-classes/libcls_log.so.1.0.0 (path inode=265675) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546092 /usr/lib64/rados-classes/libcls_sdk.so.1.0.0 (path inode=265703) 2024-02-19T19:12:05.608 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546098 /usr/lib64/rados-classes/libcls_timeindex.so.1.0.0 (path inode=265709) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546101 /usr/lib64/rados-classes/libcls_user.so.1.0.0 (path inode=265712) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546044 /usr/lib64/rados-classes/libcls_cas.so.1.0.0 (path inode=265654) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546047 /usr/lib64/rados-classes/libcls_cephfs.so.1.0.0 (path inode=265657) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546104 /usr/lib64/rados-classes/libcls_version.so.1.0.0 (path inode=265715) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546050 /usr/lib64/rados-classes/libcls_cmpomap.so.1.0.0 (path inode=265660) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546074 /usr/lib64/rados-classes/libcls_otp.so.1.0.0 (path inode=265684) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546080 /usr/lib64/rados-classes/libcls_rbd.so.1.0.0 (path inode=265691) 2024-02-19T19:12:05.609 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546056 /usr/lib64/rados-classes/libcls_hello.so.1.0.0 (path inode=265666) 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 546071 /usr/lib64/rados-classes/libcls_numops.so.1.0.0 (path inode=265681) 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 540893 /usr/lib64/ceph/compressor/libceph_snappy.so.2.0.0 (path inode=265621) 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph DEL REG 0,17 246001 /[aio] 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph DEL REG 0,17 246903 /[aio] 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 540917 /usr/lib64/ceph/erasure-code/libec_jerasure.so (path inode=265637) 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph mem REG 8,1 540927 /usr/lib64/ceph/extblkdev/libceph_ebd_vdo.so (path inode=265647) 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 0u CHR 1,3 0t0 4 /dev/null 2024-02-19T19:12:05.610 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 1w FIFO 0,13 0t0 239571 pipe 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 2w FIFO 0,13 0t0 239572 pipe 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 3r FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 4w FIFO 0,13 0t0 234408 pipe 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 5w REG 8,1 198425504 530694 /var/log/ceph/ceph-osd.5.log 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 6u a_inode 0,14 0 1076 [eventpoll:7,18,19,20,21,22,23,24,25,63,66,69,72,78,79,80,81,87,90] 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 7r FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 8w FIFO 0,13 0t0 234420 pipe 2024-02-19T19:12:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 9u a_inode 0,14 0 1076 [eventpoll:10,62,64,67,70,73,82,83,84,85,88,91] 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 10r FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 11w FIFO 0,13 0t0 234421 pipe 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 12u a_inode 0,14 0 1076 [eventpoll:13,58,65,68,71,74,75,76,77,89] 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 13r FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 14w FIFO 0,13 0t0 234422 pipe 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 15r FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 16w FIFO 0,13 0t0 242731 pipe 2024-02-19T19:12:05.612 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 17u unix 0xffffa09cb1844b00 0t0 242732 /var/run/ceph/ceph-osd.5.asok type=STREAM (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 18u IPv4 235518 0t0 TCP *:6816 (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 19u IPv4 244752 0t0 TCP *:pentbox-sim (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 20u IPv4 244771 0t0 TCP *:6818 (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 21u IPv4 244791 0t0 TCP *:6819 (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 22u IPv4 244812 0t0 TCP *:6820 (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 23u IPv4 244834 0t0 TCP *:6821 (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 24u IPv4 244857 0t0 TCP *:6822 (LISTEN) 2024-02-19T19:12:05.613 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 25u IPv4 244881 0t0 TCP *:6823 (LISTEN) 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 26r FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 27w FIFO 0,13 0t0 242760 pipe 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 28r FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 29w FIFO 0,13 0t0 242761 pipe 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 30u sock 0,8 0t0 246824 protocol: UDPv6 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 31r DIR 253,4 4096 16777359 /var/lib/ceph/osd/ceph-5 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 32uW REG 253,4 37 16777380 /var/lib/ceph/osd/ceph-5/fsid 2024-02-19T19:12:05.614 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 33u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 34u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 35u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 36u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 37u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 38u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.615 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 39u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 40u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 41u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 42u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 43u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 44u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 45u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 46u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.616 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 47u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 48u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 49u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 50u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 51u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 52u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 53u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 54u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.617 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 55u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 56u BLK 253,1 0t0 677 /dev/dm-1 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 57r FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 58u IPv4 246920 0t0 TCP smithi086.front.sepia.ceph.com:57124->smithi086.front.sepia.ceph.com:3300 (ESTABLISHED) 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 59w FIFO 0,13 0t0 246002 pipe 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 60r FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 61w FIFO 0,13 0t0 246003 pipe 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 62u IPv4 242008 0t0 TCP smithi086.front.sepia.ceph.com:52650->smithi039.front.sepia.ceph.com:6800 (ESTABLISHED) 2024-02-19T19:12:05.618 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 63u IPv4 243076 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi086.front.sepia.ceph.com:49066 (ESTABLISHED) 2024-02-19T19:12:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 64u IPv4 242068 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41140 (ESTABLISHED) 2024-02-19T19:12:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 65u IPv4 242071 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi039.front.sepia.ceph.com:34554 (ESTABLISHED) 2024-02-19T19:12:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 66u IPv4 246987 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi086.front.sepia.ceph.com:57096 (ESTABLISHED) 2024-02-19T19:12:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 67u IPv4 244002 0t0 TCP smithi086.front.sepia.ceph.com:6822->smithi039.front.sepia.ceph.com:54260 (ESTABLISHED) 2024-02-19T19:12:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 68u IPv4 241414 0t0 TCP smithi086.front.sepia.ceph.com:6820->smithi039.front.sepia.ceph.com:50152 (ESTABLISHED) 2024-02-19T19:12:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd 103318 103722 rocksdb:l ceph 69u IPv4 241417 0t0 TCP smithi086.front.sepia.ceph.com:6818->smithi086.front.sepia.ceph.com:41154 (ESTABLISHED) 2024-02-19T19:12:24.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:24.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:24.800 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:24.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:24.800 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:24.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:25.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:24.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:25.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:24.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:25.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:24.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:25.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:25.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:25.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:25.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:25.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:25.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:26.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:25.762+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:26.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:25.762+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:26.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:25.762+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:26.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:26.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:26.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:26.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:26.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:26.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:27.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:26.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:27.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:26.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:27.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:26.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:27.775 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:27.775 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:27.775 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:28.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:27.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:28.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:27.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:28.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:27.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:28.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:28.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:28.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:28.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:28.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:28.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:28.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:28.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:28.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:28.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:28.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:28.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:29.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:28.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:29.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:28.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:29.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:28.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:29.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:29.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:29.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:29.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:29.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:29.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:29.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:29.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:29.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:29.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:29.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:29.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:30.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:29.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:30.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:29.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:30.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:29.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:30.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:30.416+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:30.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:30.416+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:30.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:30.416+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:30.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:30.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:30.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:30.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:30.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:30.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:31.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:30.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:31.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:30.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:31.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:30.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:31.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:31.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:31.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:31.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:31.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:31.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:32.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:31.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:32.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:31.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:32.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:31.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:32.127 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:31.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:32.127 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:31.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:32.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:31.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:32.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:32.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:32.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:32.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:32.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:32.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:32.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:32.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:32.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:32.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:32.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:32.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:32.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:32.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:32.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:33.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:33.338+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:33.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:33.338+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:33.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:33.338+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:33.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:33.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:34.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:33.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:34.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:33.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:34.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:33.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:34.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:33.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:34.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:33.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:34.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:34.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:34.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:34.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:34.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:34.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:34.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:34.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:34.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:34.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:34.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:34.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:35.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:34.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:35.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:34.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:35.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:34.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:35.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:35.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:35.721 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:35.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:35.721 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:35.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:35.999 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:35.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:36.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:35.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:36.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:35.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:35.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:35.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:35.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:36.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:36.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:36.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:36.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:36.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:36.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:36.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:36.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:36.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:36.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:37.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:36.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:37.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:36.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:37.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:36.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:37.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:36.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:37.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:37.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:37.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:37.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:37.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:37.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:37.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:37.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:37.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:37.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:37.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:38.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:37.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:38.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:37.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:38.000 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:37.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:38.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:38.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:38.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:38.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:38.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:38.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:38.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:38.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:38.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:39.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:38.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:39.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:38.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:39.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:38.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:39.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:39.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:39.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:39.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:39.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:39.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:39.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:39.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:39.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:39.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:39.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:39.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:40.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:40.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:40.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:40.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:40.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:40.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:40.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:40.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:40.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:40.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:40.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:40.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:40.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:40.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:41.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:40.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:41.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:40.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:41.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:40.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:41.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:41.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:41.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:41.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:41.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:41.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:41.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:41.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:41.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:41.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:41.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:42.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:41.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:42.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:41.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:42.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:41.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:42.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:42.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:42.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:42.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:42.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:42.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:42.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:42.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:42.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:42.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:42.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:42.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:43.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:43.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:43.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:43.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:43.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:43.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:43.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:43.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:43.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:44.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:43.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:44.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:43.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:44.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:43.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:44.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:44.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:44.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:44.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:44.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:44.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:44.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:44.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:44.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:44.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:44.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:44.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:45.044 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:44.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:45.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:44.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:45.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:44.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:45.491 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:45.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:45.491 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:45.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:45.491 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:45.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:45.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:45.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:45.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:45.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:45.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:45.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:46.011 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:45.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:46.012 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:45.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:46.012 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:45.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:46.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:45.989+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:46.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:45.989+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:46.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:45.989+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:46.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:46.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:46.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:46.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:46.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:46.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:47.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:46.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:47.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:46.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:47.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:46.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:47.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:46.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:47.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:46.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:47.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:46.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:47.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:47.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:47.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:47.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:47.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:47.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:48.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:47.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:48.250 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:47.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:48.250 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:47.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:48.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:47.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:48.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:47.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:48.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:47.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:48.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:48.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:48.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:48.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:48.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:48.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:49.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:48.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:49.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:48.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:49.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:48.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:49.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:48.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:49.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:48.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:49.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:48.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:49.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:49.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:49.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:49.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:49.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:49.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:50.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:49.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:50.250 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:49.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:50.250 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:49.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:50.251 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:49.957+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:50.251 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:49.957+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:50.251 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:49.957+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:50.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:50.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:50.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:50.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:51.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:50.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:51.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:50.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:51.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:50.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:51.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:50.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:51.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:50.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:51.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:50.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:51.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:50.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:51.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:51.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:52.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:51.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:52.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:51.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:52.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:51.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:52.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:51.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:52.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:51.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:52.352 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:52.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:52.352 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:52.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:52.352 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:52.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:52.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:52.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:52.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:52.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:52.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:52.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:52.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:52.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:52.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:52.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:52.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:52.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:53.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:52.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:53.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:52.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:53.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:52.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:53.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:53.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:53.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:53.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:53.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:53.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:54.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:53.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:54.250 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:53.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:54.250 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:53.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:54.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:54.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:54.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:54.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:54.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:54.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:54.845 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:54.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:54.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:54.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:54.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:54.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:55.252 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:54.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:55.252 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:54.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:55.252 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:54.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:55.252 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:55.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:55.253 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:55.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:55.253 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:55.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:55.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:55.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:55.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:55.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:55.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:55.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:56.249 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:55.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:56.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:55.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:56.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:55.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:56.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:55.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:56.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:55.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:56.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:55.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:56.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:56.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:56.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:56.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:56.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:56.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:56.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:56.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:56.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:56.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:56.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:56.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:57.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:56.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:57.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:56.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:57.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:56.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:57.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:57.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:57.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:57.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:57.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:57.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:58.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:57.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:58.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:57.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:58.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:57.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:58.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:57.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:58.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:57.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:58.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:57.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:58.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:58.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:58.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:58.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:58.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:58.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:58.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:58.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:58.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:59.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:58.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:59.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:58.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:59.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:58.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:12:59.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:59.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:59.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:59.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:59.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:12:59.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:12:59.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:59.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:59.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:59.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:12:59.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:12:59.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:00.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:59.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:00.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:59.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:00.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:12:59.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:00.569+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:00.569+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:00.569+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:00.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:00.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:00.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:01.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:00.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:01.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:00.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:01.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:00.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:01.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:01.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:01.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:01.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:01.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:01.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:01.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:01.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:01.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:01.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:01.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:01.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:02.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:01.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:02.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:01.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:02.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:01.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:02.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:02.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:02.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:02.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:02.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:02.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:02.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:02.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:02.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:02.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:02.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:02.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:03.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:03.017+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:03.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:03.017+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:03.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:03.017+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:03.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:03.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:03.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:03.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:04.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:03.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:04.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:03.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:04.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:03.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:04.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:03.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:04.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:04.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:04.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:04.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:04.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:04.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:04.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:04.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:04.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:04.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:04.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:04.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:04.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:04.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:04.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:04.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:04.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:04.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:05.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:05.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:05.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:05.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:05.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:05.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:05.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:05.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:05.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:05.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:06.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:05.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:06.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:05.606+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:06.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:05.606+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:06.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:05.606+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:06.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:06.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:06.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:06.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:06.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:06.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:06.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:06.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:06.808 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:06.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:06.808 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:06.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:07.076 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:06.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:07.076 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:06.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:07.076 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:06.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:07.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:07.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:07.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:07.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:07.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:07.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:07.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:07.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:07.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:07.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:07.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:07.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:08.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:07.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:08.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:07.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:08.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:07.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:08.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:08.023+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:08.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:08.023+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:08.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:08.023+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:09.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:08.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:09.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:08.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:09.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:08.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:09.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:08.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:09.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:08.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:09.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:08.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:09.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:09.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:09.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:09.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:09.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:09.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:09.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:09.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:09.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:09.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:09.935 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:09.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:09.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:09.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:09.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:09.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:09.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:09.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:10.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:09.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:10.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:09.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:10.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:09.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:10.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:10.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:10.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:10.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:11.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:10.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:11.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:10.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:11.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:10.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:11.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:10.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:11.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:10.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:11.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:10.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:11.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:10.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:12.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:11.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:12.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:11.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:12.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:11.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:12.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:11.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:12.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:11.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:12.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:11.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:12.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:12.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:12.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:12.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:12.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:12.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:12.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:12.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:12.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:12.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:12.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:12.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:12.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:12.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:12.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:12.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:12.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:12.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:13.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:12.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:13.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:12.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:13.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:12.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:13.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:13.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:14.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:13.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:14.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:13.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:14.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:13.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:14.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:13.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:14.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:13.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:14.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:14.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:14.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:14.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:14.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:14.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:14.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:14.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:15.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:14.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:15.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:14.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:15.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:14.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:15.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:14.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:15.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:14.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:15.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:15.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:15.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:15.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:15.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:15.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:15.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:15.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:15.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:15.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:16.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:15.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:16.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:15.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:16.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:15.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:16.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:15.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:16.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:16.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:16.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:16.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:16.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:16.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:16.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:16.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:16.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:16.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:17.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:16.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:17.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:16.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:17.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:16.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:17.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:16.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:17.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:17.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:17.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:17.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:17.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:17.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:17.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:17.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:18.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:17.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:18.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:17.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:18.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:17.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:18.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:17.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:18.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:17.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:18.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:18.027+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:18.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:18.027+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:18.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:18.027+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:18.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:18.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:19.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:18.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:19.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:18.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:19.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:18.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:19.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:18.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:19.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:18.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:19.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:19.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:19.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:19.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:19.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:19.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:19.937 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:19.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:19.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:19.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:19.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:19.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:19.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:19.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:19.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:19.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:19.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:19.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:20.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:20.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:20.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:20.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:20.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:20.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:20.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:20.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:21.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:20.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:21.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:20.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:21.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:20.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:21.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:20.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:21.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:20.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:21.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:21.085+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:21.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:21.085+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:21.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:21.085+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:21.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:21.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:22.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:21.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:22.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:21.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:22.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:21.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:22.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:21.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:22.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:21.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:22.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:22.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:22.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:22.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:22.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:22.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:22.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:22.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:23.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:22.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:23.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:22.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:23.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:22.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:23.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:22.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:23.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:22.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:23.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:23.129+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:23.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:23.129+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:23.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:23.129+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:23.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:23.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:24.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:23.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:24.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:23.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:24.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:23.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:24.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:23.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:24.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:23.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:24.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:24.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:24.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:24.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:24.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:24.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:24.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:24.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:24.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:24.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:24.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:24.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:24.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:25.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:25.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:25.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:25.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:25.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:25.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:26.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:25.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:26.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:25.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:26.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:25.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:26.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:25.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:26.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:25.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:26.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:25.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:26.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:26.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:26.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:26.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:26.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:26.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:26.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:26.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:27.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:26.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:27.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:26.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:27.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:26.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:27.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:26.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:27.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:26.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:27.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:27.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:27.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:27.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:27.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:27.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:28.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:27.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:28.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:27.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:28.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:27.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:28.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:27.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:28.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:27.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:28.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:27.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:28.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:28.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:28.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:28.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:28.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:28.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:29.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:28.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:29.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:28.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:29.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:28.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:29.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:28.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:29.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:28.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:29.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:28.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:29.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:29.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:29.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:29.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:29.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:29.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:29.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:29.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:29.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:29.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:29.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:29.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:29.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:29.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:29.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:29.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:29.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:29.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:30.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:30.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:30.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:30.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:30.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:30.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:31.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:30.639+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:31.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:30.639+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:31.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:30.639+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:31.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:30.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:31.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:30.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:31.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:30.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:31.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:31.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:31.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:31.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:31.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:31.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:32.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:31.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:32.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:31.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:32.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:31.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:32.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:32.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:32.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:32.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:32.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:32.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:32.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:32.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:32.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:32.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:32.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:33.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:32.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:33.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:32.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:32.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:32.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:33.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:32.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:33.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:33.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:33.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:33.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:33.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:33.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:34.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:33.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:34.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:33.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:34.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:33.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:34.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:33.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:34.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:33.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:34.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:33.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:34.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:34.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:34.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:34.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:34.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:34.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:34.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:34.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:34.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:34.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:34.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:34.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:34.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:34.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:34.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:34.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:34.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:34.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:35.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:34.968+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:35.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:34.968+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:35.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:34.968+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:35.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:35.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:36.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:35.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:36.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:35.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:35.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:35.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:36.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:35.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:36.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:36.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:36.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:36.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:36.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:36.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:36.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:36.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:36.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:36.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:36.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:36.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:36.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:36.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:36.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:36.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:36.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:36.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:37.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:37.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:37.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:37.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:37.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:37.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:37.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:37.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:38.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:37.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:38.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:37.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:37.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:37.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:37.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:38.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:38.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:38.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:38.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:38.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:38.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:38.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:38.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:39.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:38.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:39.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:38.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:39.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:38.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:39.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:38.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:39.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:38.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:39.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:39.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:39.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:39.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:39.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:39.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:39.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:39.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:39.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:39.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:39.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:39.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:39.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:39.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:39.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:39.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:39.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:39.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:40.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:40.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:40.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:40.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:40.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:40.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:40.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:40.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:41.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:40.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:41.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:40.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:40.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:40.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:40.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:41.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:41.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:41.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:41.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:41.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:41.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:41.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:41.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:42.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:41.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:42.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:41.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:41.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:41.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:41.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:42.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:42.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:42.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:42.913 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:42.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:42.913 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:42.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:42.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:42.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:42.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:42.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:42.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:42.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:42.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:42.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:43.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:42.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:43.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:42.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:43.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:42.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:43.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:43.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:43.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:43.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:43.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:43.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:43.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:43.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:43.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:43.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:43.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:43.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:44.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:44.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:44.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:44.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:44.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:44.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:44.935 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:44.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:44.935 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:44.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:44.935 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:44.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:44.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:44.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:44.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:44.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:44.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:44.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:45.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:44.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:45.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:44.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:45.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:44.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:46.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:45.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:46.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:45.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:46.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:45.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:46.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:45.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:46.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:45.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:46.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:45.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:46.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:46.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:46.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:46.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:46.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:46.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:47.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:46.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:47.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:46.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:47.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:46.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:47.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:46.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:47.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:46.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:47.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:46.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:47.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:46.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:47.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:46.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:47.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:46.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:47.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:47.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:47.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:47.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:47.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:47.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:47.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:47.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:47.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:47.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:48.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:47.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:48.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:47.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:48.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:47.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:49.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:48.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:49.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:48.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:49.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:48.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:49.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:48.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:49.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:48.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:49.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:48.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:49.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:48.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:49.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:48.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:49.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:48.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:49.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:49.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:49.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:49.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:49.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:49.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:49.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:49.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:49.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:49.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:49.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:49.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:50.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:49.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:50.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:49.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:50.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:49.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:50.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:50.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:50.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:50.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:50.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:50.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:50.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:50.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:50.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:50.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:50.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:50.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:51.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:50.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:51.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:50.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:51.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:50.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:52.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:51.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:52.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:51.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:52.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:51.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:52.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:51.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:52.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:51.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:52.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:51.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:52.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:52.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:52.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:52.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:52.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:52.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:53.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:52.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:53.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:52.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:53.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:52.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:53.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:52.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:53.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:52.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:53.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:52.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:53.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:53.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:53.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:53.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:53.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:53.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:53.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:53.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:54.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:53.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:54.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:53.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:54.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:53.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:54.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:53.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:54.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:53.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:54.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:54.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:54.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:54.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:54.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:54.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:54.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:54.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:54.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:54.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:54.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:54.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:54.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:54.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:54.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:54.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:54.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:54.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:55.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:55.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:55.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:55.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:55.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:55.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:55.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:55.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:55.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:55.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:55.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:55.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:55.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:55.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:55.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:55.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:55.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:55.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:56.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:55.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:56.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:55.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:56.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:55.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:56.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:56.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:57.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:56.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:57.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:56.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:57.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:56.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:57.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:56.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:57.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:56.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:57.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:56.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:57.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:56.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:57.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:56.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:57.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:57.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:57.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:57.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:57.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:57.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:57.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:57.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:57.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:57.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:57.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:57.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:58.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:57.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:58.251 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:57.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:58.251 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:57.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:58.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:58.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:58.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:58.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:58.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:58.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:58.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:58.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:58.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:58.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:58.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:58.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:59.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:58.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:59.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:58.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:59.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:58.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:13:59.937 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:59.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:59.937 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:59.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:59.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:13:59.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:13:59.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:59.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:59.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:59.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:13:59.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:13:59.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:00.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:59.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:00.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:59.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:00.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:13:59.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:00.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:00.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:00.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:00.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:00.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:00.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:00.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:00.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:00.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:00.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:00.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:00.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:01.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:01.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:01.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:01.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:01.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:02.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:01.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:02.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:01.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:02.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:01.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:02.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:01.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:02.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:01.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:02.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:01.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:02.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:01.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:02.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:01.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:02.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:02.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:02.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:02.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:02.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:02.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:02.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:02.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:02.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:02.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:02.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:02.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:03.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:02.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:03.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:02.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:03.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:02.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:03.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:03.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:03.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:03.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:03.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:03.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:03.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:03.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:03.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:03.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:03.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:03.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:04.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:03.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:04.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:03.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:04.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:03.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:04.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:04.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:04.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:04.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:04.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:04.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:04.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:04.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:04.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:04.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:04.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:04.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:05.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:04.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:05.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:04.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:05.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:04.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:05.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:05.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:05.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:05.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:05.644 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:05.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:05.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:05.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:05.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:05.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:05.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:05.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:06.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:05.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:06.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:05.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:06.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:05.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:06.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:06.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:06.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:06.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:06.613+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:06.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:06.613+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:06.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:06.613+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:07.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:06.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:07.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:06.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:07.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:06.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:07.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:07.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:07.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:07.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:07.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:07.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:07.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:07.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:07.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:07.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:07.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:07.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:08.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:07.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:08.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:07.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:08.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:07.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:08.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:08.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:08.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:08.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:08.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:08.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:08.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:08.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:08.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:08.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:08.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:08.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:09.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:08.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:09.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:08.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:09.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:08.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:09.928 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:09.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:09.928 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:09.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:09.929 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:09.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:09.929 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:09.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:09.929 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:09.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:09.929 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:09.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:10.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:09.904+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:10.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:09.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:10.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:09.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:10.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:09.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:10.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:10.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:10.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:10.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:10.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:10.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:10.964 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:10.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:10.964 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:10.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:10.965 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:10.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:11.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:10.940+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:11.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:10.940+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:11.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:10.940+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:11.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:11.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:11.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:11.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:11.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:11.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:11.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:11.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:11.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:11.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:11.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:11.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:12.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:11.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:12.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:11.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:12.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:11.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:12.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:12.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:12.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:12.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:12.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:12.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:12.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:12.547+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:12.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:12.547+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:12.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:12.547+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:13.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:12.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:13.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:12.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:13.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:12.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:13.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:13.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:13.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:13.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:13.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:13.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:13.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:13.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:13.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:13.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:13.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:13.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:14.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:13.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:14.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:13.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:14.251 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:13.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:14.929 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:14.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:14.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:14.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:14.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:14.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:14.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:14.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:14.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:14.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:14.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:14.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:15.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:14.905+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:15.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:14.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:15.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:14.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:15.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:14.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:16.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:15.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:16.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:15.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:16.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:15.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:16.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:15.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:16.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:15.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:16.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:15.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:16.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:15.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:16.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:15.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:16.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:15.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:16.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:16.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:16.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:16.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:16.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:16.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:16.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:16.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:16.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:16.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:16.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:16.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:17.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:16.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:17.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:16.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:17.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:16.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:17.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:17.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:17.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:17.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:17.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:17.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:17.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:17.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:17.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:17.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:17.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:17.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:18.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:17.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:18.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:17.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:18.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:17.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:18.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:18.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:18.945 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:18.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:18.945 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:18.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:18.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:18.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:18.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:19.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:18.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:19.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:18.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:19.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:18.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:19.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:19.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:19.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:19.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:19.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:19.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:19.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:19.540+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:19.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:19.540+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:19.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:19.540+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:20.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:19.905+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:20.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:19.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:20.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:19.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:20.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:19.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:20.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:20.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:20.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:20.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:20.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:20.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:20.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:20.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:20.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:20.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:20.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:20.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:21.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:20.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:21.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:20.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:21.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:20.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:21.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:21.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:21.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:21.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:21.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:21.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:21.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:21.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:21.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:22.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:21.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:22.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:21.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:22.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:21.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:22.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:22.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:22.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:22.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:22.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:22.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:23.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:22.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:23.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:22.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:23.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:22.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:23.619 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:23.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:23.619 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:23.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:23.619 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:23.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:24.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:23.595+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:24.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:23.595+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:24.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:23.595+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:24.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:23.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:24.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:23.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:24.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:23.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:24.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:24.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:24.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:24.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:24.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:24.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:24.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:24.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:24.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:24.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:24.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:24.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:25.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:24.906+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 100 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:25.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:25.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:25.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:25.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:25.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:25.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:26.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:25.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:26.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:25.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:26.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:25.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:26.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:25.495+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:26.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:25.495+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:26.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:25.495+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:26.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:26.047+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:26.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:26.047+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:26.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:26.047+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:26.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:26.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:26.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:26.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:26.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:26.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:26.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:26.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:26.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:26.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:26.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:26.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:27.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:27.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:27.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:27.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:27.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:27.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:28.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:27.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:28.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:27.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:28.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:27.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:28.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:27.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:28.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:27.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:28.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:27.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:28.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:28.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:28.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:28.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:28.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:28.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:28.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:28.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:28.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:28.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:28.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:28.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:28.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:28.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:28.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:28.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:28.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:28.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:29.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:29.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:29.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:29.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:29.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:29.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:29.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:29.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:29.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:29.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:29.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:29.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:29.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:29.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:29.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:29.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:29.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:29.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:30.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:29.907+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 128 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:30.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:30.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:30.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:30.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:30.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:30.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:31.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:30.512+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:31.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:30.512+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:31.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:30.512+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:31.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:30.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:31.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:30.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:31.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:30.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:31.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:31.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:31.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:31.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:31.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:31.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:32.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:31.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:32.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:31.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:32.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:31.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:32.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:31.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:32.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:31.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:32.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:31.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:32.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:32.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:32.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:32.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:32.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:32.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:32.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:32.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:32.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:33.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:32.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:33.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:32.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:33.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:32.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:33.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:32.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:33.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:33.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:33.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:33.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:33.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:33.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:33.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:33.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:33.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:33.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:33.750 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:33.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:33.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:33.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:33.750 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:33.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:33.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:33.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:34.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:34.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:34.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:34.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:34.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:34.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:34.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:34.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:34.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:34.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:34.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:34.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:34.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:34.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:34.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:34.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:34.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:34.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:35.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:34.908+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 155 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:35.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:35.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:35.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:35.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:35.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:35.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:35.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:35.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:36.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:35.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:36.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:35.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:35.515+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:35.515+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:36.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:35.515+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:36.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:36.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:36.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:36.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:36.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:36.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:36.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:36.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:36.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:36.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:36.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:36.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:36.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:36.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:36.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:36.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:36.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:36.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:37.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:37.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:37.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:37.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:37.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:37.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:37.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:37.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:38.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:37.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:38.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:37.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:37.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:37.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:38.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:37.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:38.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:38.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:38.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:38.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:38.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:38.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:38.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:38.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:39.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:38.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:39.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:38.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:39.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:38.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:39.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:38.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:39.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:38.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:39.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:39.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:39.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:39.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:39.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:39.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:39.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:39.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:39.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:39.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:39.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:39.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:39.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:39.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:39.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:39.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:39.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:39.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:40.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:39.909+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 183 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:40.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:40.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:40.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:40.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:40.250 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:40.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:40.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:40.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:41.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:40.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:41.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:40.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:40.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:40.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:41.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:40.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:41.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:41.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:41.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:41.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:41.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:41.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:41.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:41.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:41.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:41.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:41.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:41.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:41.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:41.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:42.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:41.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:42.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:42.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:42.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:42.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:42.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:42.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:42.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:42.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:43.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:42.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:43.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:42.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:43.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:42.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:43.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:42.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:43.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:42.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:43.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:43.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:43.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:43.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:43.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:43.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:43.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:43.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:44.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:43.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:44.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:43.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:44.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:43.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:44.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:43.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:44.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:43.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:44.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:44.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:44.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:44.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:44.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:44.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:44.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:44.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:44.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:44.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:44.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:44.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:44.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:44.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:44.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:44.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:44.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:44.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:45.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:44.910+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 210 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:45.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:45.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:45.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:45.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:45.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:45.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:45.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:45.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:45.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:45.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:45.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:45.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:45.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:45.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:46.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:45.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:46.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:45.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:46.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:46.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:46.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:46.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:46.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:46.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:46.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:46.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:46.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:46.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:46.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:46.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:47.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:46.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:47.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:46.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:47.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:46.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:47.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:47.135+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:47.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:47.135+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:47.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:47.135+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:47.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:47.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:47.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:47.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:47.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:47.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:48.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:47.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:48.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:47.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:48.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:47.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:48.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:48.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:48.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:48.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:48.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:48.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:48.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:48.519+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:48.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:48.519+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:49.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:48.519+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:49.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:48.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:49.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:48.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:49.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:48.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:49.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:49.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:49.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:49.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:49.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:49.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:49.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:49.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:49.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:49.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:49.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:49.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:49.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:49.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:49.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:49.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:49.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:49.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:50.186 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:49.910+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 238 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:50.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:50.164+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:50.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:50.164+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:50.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:50.164+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:50.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:50.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:50.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:50.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:50.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:50.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:51.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:50.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:51.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:50.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:51.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:50.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:51.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:51.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:51.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:51.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:51.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:51.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:51.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:51.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:51.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:51.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:51.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:51.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:51.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:51.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:51.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:51.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:52.000 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:51.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:52.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:52.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:52.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:52.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:52.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:52.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:52.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:52.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:52.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:52.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:52.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:52.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:53.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:52.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:53.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:52.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:53.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:52.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:53.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:53.185+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:53.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:53.185+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:53.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:53.185+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:53.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:53.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:53.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:53.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:53.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:53.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:53.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:53.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:53.999 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:53.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:54.000 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:53.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:54.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:54.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:54.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:54.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:54.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:54.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:54.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:54.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:54.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:54.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:54.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:54.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:54.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:54.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:54.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:54.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:54.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:54.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:55.207 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:54.911+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 265 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:14:55.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:55.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:55.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:55.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:55.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:55.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:55.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:55.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:55.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:55.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:55.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:55.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:56.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:55.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:56.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:55.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:56.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:55.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:56.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:56.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:56.450 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:56.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:56.450 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:56.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:56.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:56.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:56.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:56.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:56.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:56.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:57.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:56.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:57.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:56.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:57.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:56.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:57.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:57.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:57.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:57.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:57.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:57.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:57.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:57.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:57.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:57.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:57.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:57.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:58.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:57.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:58.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:57.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:58.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:57.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:58.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:58.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:58.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:58.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:58.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:58.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:58.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:58.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:58.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:58.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:58.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:58.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:59.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:58.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:59.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:58.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:59.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:58.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:14:59.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:59.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:59.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:59.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:59.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:14:59.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:14:59.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:59.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:59.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:59.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:14:59.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:14:59.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:00.062 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:14:59.911+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 293 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:00.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:59.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:00.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:59.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:00.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:14:59.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:00.354 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:00.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:00.354 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:00.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:00.354 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:00.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:00.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:00.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:00.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:00.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:00.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:00.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:01.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:00.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:01.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:00.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:01.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:00.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:01.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:01.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:01.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:01.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:01.500 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:01.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:01.500 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:01.324+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:01.500 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:01.324+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:01.500 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:01.324+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:01.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:01.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:01.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:01.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:01.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:01.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:02.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:02.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:02.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:02.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:02.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:02.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:02.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:02.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:02.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:02.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:02.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:02.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:03.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:02.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:03.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:02.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:03.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:02.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:03.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:03.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:03.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:03.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:03.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:03.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:03.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:03.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:03.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:03.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:03.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:03.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:04.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:03.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:04.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:03.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:04.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:03.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:04.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:04.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:04.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:04.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:04.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:04.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:04.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:04.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:04.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:04.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:04.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:04.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:05.061 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:04.911+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 320 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:05.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:04.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:05.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:04.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:05.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:04.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:05.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:04.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:05.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:05.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:05.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:05.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:05.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:05.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:05.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:05.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:05.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:05.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:05.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:05.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:06.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:05.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:06.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:05.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:06.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:05.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:06.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:05.766+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:06.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:06.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:06.460 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:06.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:06.460 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:06.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:06.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:06.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:06.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:06.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:06.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:06.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:07.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:06.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:07.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:06.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:07.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:06.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:07.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:06.738+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:07.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:07.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:07.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:07.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:07.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:07.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:07.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:07.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:07.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:07.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:07.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:07.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:08.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:07.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:08.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:07.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:08.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:07.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:08.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:07.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:08.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:08.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:08.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:08.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:08.499 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:08.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:08.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:08.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:08.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:08.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:08.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:08.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:09.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:08.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:09.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:08.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:09.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:08.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:09.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:08.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:09.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:09.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:09.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:09.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:09.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:09.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:09.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:09.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:09.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:09.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:09.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:09.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:10.061 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:09.912+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 348 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:10.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:09.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:10.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:09.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:10.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:09.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:10.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:09.733+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:10.388 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:10.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:10.388 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:10.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:10.388 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:10.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:10.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:10.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:10.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:10.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:10.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:10.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:10.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:10.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:10.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:10.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:10.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:10.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:10.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:10.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:11.409 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:11.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:11.409 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:11.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:11.409 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:11.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:11.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:11.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:11.689 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:11.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:11.689 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:11.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:11.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:11.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:11.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:11.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:11.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:11.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:11.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:11.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:12.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:12.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:12.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:12.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:12.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:12.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:12.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:12.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:12.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:12.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:12.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:12.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:12.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:12.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:12.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:12.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:12.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:12.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:12.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:12.639+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:13.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:13.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:13.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:13.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:13.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:13.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:13.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:13.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:13.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:13.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:13.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:13.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:13.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:13.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:13.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:13.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:13.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:13.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:13.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:13.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:14.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:14.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:14.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:14.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:14.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:14.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:14.672 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:14.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:14.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:14.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:14.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:14.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:14.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:14.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:14.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:14.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:14.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:14.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:14.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:14.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:15.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:14.913+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 375 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:15.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:15.084+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:15.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:15.084+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:15.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:15.084+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:15.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:15.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:15.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:15.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:15.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:15.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:15.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:15.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:15.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:15.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:15.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:15.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:15.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:15.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:16.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:16.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:16.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:16.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:16.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:16.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:16.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:16.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:16.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:16.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:16.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:16.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:16.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:16.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:16.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:16.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:16.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:16.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:16.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:16.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:17.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:17.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:17.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:17.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:17.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:17.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:17.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:17.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:17.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:17.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:17.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:17.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:17.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:17.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:17.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:17.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:17.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:17.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:18.429 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:18.077+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:18.429 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:18.077+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:18.429 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:18.077+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:18.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:18.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:18.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:18.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:18.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:18.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:18.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:18.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:18.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:18.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:18.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:18.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:18.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:18.715+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:19.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:19.128+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:19.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:19.128+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:19.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:19.128+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:19.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:19.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:19.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:19.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:19.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:19.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:19.998 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:19.914+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 402 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:19.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:19.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:19.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:19.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:19.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:19.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:19.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:19.718+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:20.445 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:20.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:20.445 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:20.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:20.446 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:20.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:20.715 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:20.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:20.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:20.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:20.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:20.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:20.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:20.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:20.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:20.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:20.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:20.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:20.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:20.694+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:21.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:21.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:21.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:21.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:21.498 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:21.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:21.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:21.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:21.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:21.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:21.499 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:21.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:21.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:21.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:21.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:21.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:21.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:21.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:21.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:21.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:22.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:22.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:22.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:22.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:22.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:22.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:22.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:22.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:22.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:22.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:22.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:22.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:22.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:22.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:22.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:22.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:22.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:22.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:22.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:22.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:23.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:23.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:23.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:23.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:23.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:23.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:23.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:23.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:23.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:23.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:23.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:23.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:23.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:23.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:23.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:23.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:23.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:23.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:23.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:23.598+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:24.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:24.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:24.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:24.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:24.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:24.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:24.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:24.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:24.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:24.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:24.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:24.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:24.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:24.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:24.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:24.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:24.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:24.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:24.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:24.575+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:25.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:24.914+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 430 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:25.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:25.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:25.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:25.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:25.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:25.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:25.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:25.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:25.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:25.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:25.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:25.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:25.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:25.616+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:25.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:25.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:25.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:25.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:25.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:25.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:26.496 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:26.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:26.496 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:26.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:26.496 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:26.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:26.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:26.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:26.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:26.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:26.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:26.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:26.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:26.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:26.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:26.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:26.748 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:26.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:26.749 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:26.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:27.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:27.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:27.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:27.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:27.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:27.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:27.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:27.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:27.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:27.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:27.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:27.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:27.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:27.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:27.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:27.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:27.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:27.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:27.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:27.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:28.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:28.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:28.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:28.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:28.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:28.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:28.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:28.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:28.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:28.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:28.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:28.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:28.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:28.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:28.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:28.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:28.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:28.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:28.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:28.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:29.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:29.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:29.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:29.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:29.498 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:29.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:29.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:29.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:29.937 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:29.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:29.937 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:29.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:29.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:29.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:29.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:29.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:29.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:29.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:29.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:29.718+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:30.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:29.916+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 457 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:30.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:30.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:30.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:30.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:30.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:30.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:30.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:30.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:30.997 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:30.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:30.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:30.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:30.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:30.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:30.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:30.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:30.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:30.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:30.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:30.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:31.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:31.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:31.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:31.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:31.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:31.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:31.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:31.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:31.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:31.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:31.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:31.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:32.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:31.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:32.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:31.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:32.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:31.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:32.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:31.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:32.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:32.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:32.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:32.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:32.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:32.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:32.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:32.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:32.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:32.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:32.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:32.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:33.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:32.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:33.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:32.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:33.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:32.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:33.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:32.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:33.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:33.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:33.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:33.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:33.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:33.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:33.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:33.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:33.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:33.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:33.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:33.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:33.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:33.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:33.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:33.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:33.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:33.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:33.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:33.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:34.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:34.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:34.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:34.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:34.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:34.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:34.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:34.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:34.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:34.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:34.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:34.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:34.948 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:34.916+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 489 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:34.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:34.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:34.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:34.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:34.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:34.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:34.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:34.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:35.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:35.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:35.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:35.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:35.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:35.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:35.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:35.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:35.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:35.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:35.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:35.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:35.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:35.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:35.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:35.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:35.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:35.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:35.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:35.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:36.411 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:36.088+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:36.411 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:36.088+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:36.411 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:36.088+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:36.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:36.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:36.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:36.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:36.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:36.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:36.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:36.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:36.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:36.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:36.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:36.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:36.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:36.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:37.402 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:37.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:37.402 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:37.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:37.403 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:37.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:37.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:37.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:37.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:37.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:37.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:37.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:37.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:37.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:37.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:37.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:37.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:37.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:37.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:37.714+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:38.413 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:38.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:38.413 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:38.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:38.413 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:38.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:38.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:38.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:38.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:38.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:38.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:38.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:39.012 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:38.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:39.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:38.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:39.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:38.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:39.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:38.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:39.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:38.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:39.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:38.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:39.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:38.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:39.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:39.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:39.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:39.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:39.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:39.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:40.061 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:39.917+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 516 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:40.061 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:39.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:40.061 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:39.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:40.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:39.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:40.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:39.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:40.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:39.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:40.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:39.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:40.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:39.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:40.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:40.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:40.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:40.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:40.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:40.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:41.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:40.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:41.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:40.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:41.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:40.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:41.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:40.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:41.390 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:41.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:41.391 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:41.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:41.391 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:41.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:41.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:41.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:41.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:41.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:41.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:41.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:42.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:41.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:42.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:41.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:42.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:41.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:42.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:41.774+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:42.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:42.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:42.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:42.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:42.380 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:42.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:42.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:42.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:42.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:42.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:42.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:42.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:43.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:43.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:43.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:43.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:43.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:43.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:43.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:42.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:43.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:42.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:43.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:42.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:43.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:42.801+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:43.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:43.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:43.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:43.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:43.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:43.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:44.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:44.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:44.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:44.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:44.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:44.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:44.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:43.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:44.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:43.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:44.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:43.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:44.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:43.845+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:44.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:44.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:44.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:44.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:44.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:44.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:45.247 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:44.918+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 544 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:45.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:44.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:45.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:44.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:45.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:44.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:45.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:44.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:45.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:45.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:45.248 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:45.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:45.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:45.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:45.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:45.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:45.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:45.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:45.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:45.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:46.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:46.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:46.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:46.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:46.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:46.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:46.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:45.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:46.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:45.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:46.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:45.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:46.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:45.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:46.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:46.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:46.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:46.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:46.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:46.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:47.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:46.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:47.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:46.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:47.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:46.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:47.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:46.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:47.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:46.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:47.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:46.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:47.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:46.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:47.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:47.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:47.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:47.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:47.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:47.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:48.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:47.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:48.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:47.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:48.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:47.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:48.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:47.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:48.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:47.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:48.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:47.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:48.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:47.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:48.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:48.292+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:48.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:48.292+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:48.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:48.292+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:49.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:48.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:49.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:48.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:49.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:48.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:49.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:48.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:49.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:48.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:49.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:48.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:49.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:48.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:49.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:49.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:49.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:49.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:49.747 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:49.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:50.066 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:49.917+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 574 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:50.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:49.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:50.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:49.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:50.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:49.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:50.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:49.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:50.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:49.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:50.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:49.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:50.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:49.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:50.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:50.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:50.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:50.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:50.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:50.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:51.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:50.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:51.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:50.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:51.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:50.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:51.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:50.733+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:51.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:50.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:51.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:50.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:51.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:50.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:51.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:51.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:51.744 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:51.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:51.744 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:51.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:51.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:51.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:51.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:51.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:51.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:51.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:51.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:51.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:51.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:51.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:51.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:51.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:51.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:51.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:52.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:52.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:52.704 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:52.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:52.704 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:52.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:52.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:52.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:52.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:52.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:52.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:52.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:52.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:52.683+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:52.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:52.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:52.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:52.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:52.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:52.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:53.566 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:53.250+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:53.567 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:53.250+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:53.567 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:53.250+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:53.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:53.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:53.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:53.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:53.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:53.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:53.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:53.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:54.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:53.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:54.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:53.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:54.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:53.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:54.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:54.210+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:54.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:54.210+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:54.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:54.210+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:55.065 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:54.918+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 605 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:15:55.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:54.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:55.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:54.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:55.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:54.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:55.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:54.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:55.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:54.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:55.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:54.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:55.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:54.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:55.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:55.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:55.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:55.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:55.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:55.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:56.001 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:55.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:56.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:55.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:56.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:55.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:56.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:55.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:56.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:56.180+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:56.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:56.180+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:56.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:56.180+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:56.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:55.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:56.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:55.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:56.497 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:55.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:57.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:56.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:57.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:56.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:57.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:56.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:57.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:56.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:57.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:56.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:57.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:56.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:57.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:56.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:57.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:57.135+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:57.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:57.135+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:57.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:57.135+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:58.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:57.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:58.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:57.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:58.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:57.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:58.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:57.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:58.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:57.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:58.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:57.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:58.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:57.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:58.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:58.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:58.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:58.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:58.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:58.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:59.082 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:58.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:59.083 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:58.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:59.083 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:58.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:15:59.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:58.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:59.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:58.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:59.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:58.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:15:59.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:58.747+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:59.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:59.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:59.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:59.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:15:59.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:15:59.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:00.067 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:15:59.919+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 632 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:00.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:59.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:00.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:59.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:00.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:15:59.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:00.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:59.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:00.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:59.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:00.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:59.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:00.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:15:59.782+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:00.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:00.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:00.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:00.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:00.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:00.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:01.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:00.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:01.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:00.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:01.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:00.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:01.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:00.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:01.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:00.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:01.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:00.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:01.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:00.814+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:01.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:01.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:01.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:01.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:01.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:01.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:02.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:01.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:02.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:01.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:02.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:01.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:02.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:01.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:02.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:01.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:02.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:01.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:02.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:01.825+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:02.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:02.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:02.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:02.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:02.497 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:02.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:03.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:02.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:03.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:02.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:03.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:02.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:03.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:02.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:03.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:02.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:03.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:02.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:03.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:02.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:03.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:03.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:03.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:03.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:03.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:03.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:04.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:03.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:04.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:03.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:04.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:03.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:04.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:03.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:04.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:03.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:04.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:03.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:04.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:03.799+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:04.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:04.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:04.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:04.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:04.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:04.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:05.068 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:04.920+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 660 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:05.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:04.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:05.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:04.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:05.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:04.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:05.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:04.755+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:05.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:04.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:05.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:04.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:05.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:04.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:05.382 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:05.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:05.382 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:05.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:05.382 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:05.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:06.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:05.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:06.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:05.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:06.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:05.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:05.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:06.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:05.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:06.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:05.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:06.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:05.799+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:06.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:06.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:06.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:06.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:06.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:06.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:07.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:06.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:07.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:06.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:07.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:06.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:07.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:06.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:07.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:06.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:07.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:06.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:07.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:06.839+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:07.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:07.230+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:07.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:07.230+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:07.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:07.230+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:08.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:08.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:08.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:08.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:07.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:08.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:07.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:08.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:07.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:08.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:07.861+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:08.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:08.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:08.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:08.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:08.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:08.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:09.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:08.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:09.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:08.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:09.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:08.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:09.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:08.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:09.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:08.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:09.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:08.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:09.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:08.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:09.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:09.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:09.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:09.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:09.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:09.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:09.886 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:09.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:09.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:09.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:09.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:09.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:10.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:09.921+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 687 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:10.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:09.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:10.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:09.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:10.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:09.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:10.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:09.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:10.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:10.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:10.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:10.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:10.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:10.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:11.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:10.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:11.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:10.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:11.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:10.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:11.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:10.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:11.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:10.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:11.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:10.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:11.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:10.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:11.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:11.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:11.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:11.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:11.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:11.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:12.203 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:11.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:12.203 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:11.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:12.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:11.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:12.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:11.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:12.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:11.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:12.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:11.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:12.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:11.825+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:12.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:12.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:12.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:12.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:12.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:12.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:13.243 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:12.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:13.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:12.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:13.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:12.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:13.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:12.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:13.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:12.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:13.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:12.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:13.245 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:12.832+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:13.495 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:13.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:13.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:13.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:13.496 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:13.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:13.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:13.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:13.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:13.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:13.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:13.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:13.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:13.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:13.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:13.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:13.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:13.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:13.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:13.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:14.593 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:14.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:14.593 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:14.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:14.593 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:14.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:14.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:14.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:14.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:14.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:14.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:14.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:14.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:14.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:14.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:14.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:14.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:14.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:14.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:14.758+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:15.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:14.921+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 715 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:15.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:15.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:15.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:15.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:15.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:15.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:16.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:15.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:16.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:15.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:16.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:15.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:16.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:15.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:16.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:15.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:16.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:15.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:16.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:15.763+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:16.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:16.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:16.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:16.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:16.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:16.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:17.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:16.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:17.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:16.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:17.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:16.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:17.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:16.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:17.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:16.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:17.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:16.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:17.247 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:16.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:17.715 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:17.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:17.715 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:17.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:17.715 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:17.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:17.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:17.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:17.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:17.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:17.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:17.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:17.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:17.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:17.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:17.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:17.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:17.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:17.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:17.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:18.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:18.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:18.687 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:18.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:18.687 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:18.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:18.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:18.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:18.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:18.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:18.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:18.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:18.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:18.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:18.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:18.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:18.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:18.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:18.997 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:18.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:19.689 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:19.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:19.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:19.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:19.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:19.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:19.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:19.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:19.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:19.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:19.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:19.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:19.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:19.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:19.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:19.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:19.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:19.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:19.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:19.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:20.245 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:19.923+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 742 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:20.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:20.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:20.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:20.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:20.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:20.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:20.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:20.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:20.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:20.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:20.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:20.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:20.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:20.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:20.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:20.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:20.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:20.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:20.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:20.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:21.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:21.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:21.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:21.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:21.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:21.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:21.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:21.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:21.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:21.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:21.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:21.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:21.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:21.593+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:22.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:21.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:22.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:21.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:22.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:21.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:22.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:22.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:22.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:22.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:22.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:22.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:22.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:22.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:22.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:22.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:22.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:22.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:22.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:22.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:23.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:22.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:23.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:22.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:23.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:22.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:23.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:23.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:23.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:23.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:23.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:23.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:23.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:23.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:23.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:23.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:23.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:23.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:23.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:23.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:24.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:23.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:24.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:23.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:24.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:23.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:24.705 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:24.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:24.705 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:24.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:24.705 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:24.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:24.965 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:24.923+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 771 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:24.966 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:24.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:24.966 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:24.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:24.966 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:24.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:24.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:24.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:24.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:24.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:24.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:24.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:24.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:24.697+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:25.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:25.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:25.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:25.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:25.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:25.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:25.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:25.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:25.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:25.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:25.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:25.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:25.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:25.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:25.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:25.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:25.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:25.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:25.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:25.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:26.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:26.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:26.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:26.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:26.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:26.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:26.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:26.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:26.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:26.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:26.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:26.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:26.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:26.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:26.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:26.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:26.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:26.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:26.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:26.714+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:27.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:27.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:27.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:27.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:27.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:27.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:28.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:27.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:28.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:27.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:28.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:27.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:28.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:27.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:28.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:27.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:28.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:27.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:28.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:27.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:28.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:28.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:28.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:28.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:28.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:28.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:29.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:28.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:29.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:28.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:29.246 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:28.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:29.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:28.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:29.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:28.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:29.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:28.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:29.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:28.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:29.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:29.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:29.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:29.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:29.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:29.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:30.069 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:29.924+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 798 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:30.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:29.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:30.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:29.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:30.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:29.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:30.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:29.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:30.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:29.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:30.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:29.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:30.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:29.792+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:30.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:30.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:30.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:30.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:30.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:30.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:31.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:30.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:31.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:30.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:31.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:30.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:31.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:30.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:31.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:30.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:31.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:30.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:31.246 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:30.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:31.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:31.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:31.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:31.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:31.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:31.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:31.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:31.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:31.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:31.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:31.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:31.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:31.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:31.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:31.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:31.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:31.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:31.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:31.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:31.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:32.702 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:32.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:32.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:32.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:32.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:32.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:32.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:32.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:32.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:32.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:32.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:32.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:32.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:32.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:32.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:32.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:32.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:32.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:32.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:32.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:33.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:33.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:33.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:33.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:33.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:33.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:33.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:33.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:33.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:33.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:33.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:33.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:33.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:33.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:34.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:33.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:34.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:33.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:34.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:33.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:34.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:34.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:34.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:34.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:34.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:34.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:34.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:34.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:34.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:34.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:34.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:34.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:34.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:34.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:35.069 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:34.925+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 826 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:35.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:34.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:35.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:34.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:35.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:34.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:35.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:35.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:35.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:35.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:35.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:35.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:35.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:35.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:35.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:35.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:35.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:35.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:35.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:35.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:36.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:35.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:36.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:35.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:36.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:35.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:36.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:36.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:36.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:36.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:36.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:36.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:36.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:36.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:36.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:36.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:36.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:36.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:36.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:36.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:37.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:36.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:37.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:36.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:37.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:36.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:37.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:37.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:37.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:37.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:37.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:37.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:37.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:37.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:37.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:37.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:37.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:37.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:37.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:37.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:37.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:37.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:37.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:37.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:37.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:37.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:38.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:38.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:38.709 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:38.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:38.709 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:38.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:38.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:38.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:38.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:38.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:38.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:38.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:38.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:38.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:38.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:38.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:38.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:38.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:38.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:38.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:39.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:39.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:39.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:39.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:39.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:39.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:39.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:39.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:39.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:39.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:39.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:39.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:39.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:39.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:40.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:39.925+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 853 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:40.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:39.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:40.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:39.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:40.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:39.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:40.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:40.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:40.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:40.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:40.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:40.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:40.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:40.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:40.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:40.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:40.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:40.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:40.845 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:40.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:41.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:40.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:41.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:40.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:41.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:40.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:41.825 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:41.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:41.826 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:41.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:41.826 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:41.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:41.826 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:41.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:41.826 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:41.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:41.826 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:41.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:41.826 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:41.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:42.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:41.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:42.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:41.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:42.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:41.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:42.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:42.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:42.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:42.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:42.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:42.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:42.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:42.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:42.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:42.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:42.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:42.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:42.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:42.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:43.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:42.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:43.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:42.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:43.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:42.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:43.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:43.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:43.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:43.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:43.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:43.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:43.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:43.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:43.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:43.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:43.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:43.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:43.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:43.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:44.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:43.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:44.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:43.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:44.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:43.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:44.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:44.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:44.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:44.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:44.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:44.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:44.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:44.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:44.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:44.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:44.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:44.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:44.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:44.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:45.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:44.925+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 881 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:45.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:44.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:45.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:44.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:45.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:44.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:45.908 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:45.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:45.909 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:45.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:45.909 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:45.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:45.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:45.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:45.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:45.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:45.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:45.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:45.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:45.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:46.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:45.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:46.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:45.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:46.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:45.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:46.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:46.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:46.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:46.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:46.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:46.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:46.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:46.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:46.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:46.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:46.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:46.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:46.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:46.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:46.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:46.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:46.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:46.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:46.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:46.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:47.917 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:47.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:47.918 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:47.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:47.918 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:47.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:47.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:47.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:47.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:47.919 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:47.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:48.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:47.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:48.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:47.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:48.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:47.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:48.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:48.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:48.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:48.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:48.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:48.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:48.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:48.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:48.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:48.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:48.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:48.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:48.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:48.739+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:48.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:48.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:48.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:48.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:48.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:48.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:49.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:49.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:49.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:49.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:49.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:49.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:49.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:49.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:49.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:49.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:49.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:49.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:49.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:49.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:49.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:49.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:49.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:49.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:49.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:49.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:50.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:49.926+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 912 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:50.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:50.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:50.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:50.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:50.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:50.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:50.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:50.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:50.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:50.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:50.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:50.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:50.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:50.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:50.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:50.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:50.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:51.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:51.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:51.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:51.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:51.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:51.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:51.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:51.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:51.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:51.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:51.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:51.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:51.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:51.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:51.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:51.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:51.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:51.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:51.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:51.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:52.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:52.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:52.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:52.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:52.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:52.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:52.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:52.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:52.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:52.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:52.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:52.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:52.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:52.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:52.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:52.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:52.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:52.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:52.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:52.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:53.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:53.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:53.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:53.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:53.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:53.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:53.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:53.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:53.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:53.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:53.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:53.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:53.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:53.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:53.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:53.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:53.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:53.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:53.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:53.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:54.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:54.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:54.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:54.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:54.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:54.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:54.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:54.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:54.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:54.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:54.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:54.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:54.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:54.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:55.244 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:54.927+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 940 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:55.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:54.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:55.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:54.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:55.245 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:54.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:55.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:55.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:55.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:55.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:55.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:55.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:55.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:55.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:55.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:55.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:55.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:55.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:55.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:55.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:55.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:55.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:55.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:55.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:55.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:55.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:56.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:56.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:56.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:56.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:56.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:56.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:56.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:56.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:56.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:56.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:56.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:56.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:56.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:56.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:56.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:56.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:56.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:56.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:56.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:56.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:57.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:57.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:57.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:57.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:57.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:57.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:57.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:57.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:57.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:57.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:57.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:57.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:57.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:57.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:57.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:57.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:57.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:57.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:57.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:57.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:58.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:58.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:58.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:58.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:58.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:58.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:58.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:58.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:58.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:58.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:58.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:58.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:58.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:58.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:58.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:58.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:58.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:58.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:58.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:58.748+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:59.838 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-02-19T19:16:59.968 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:16:59.928+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 967 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:16:59.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:59.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:59.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:59.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:59.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:16:59.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:16:59.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:59.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:59.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:59.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:59.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:16:59.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:16:59.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:59.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:59.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:59.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:59.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:59.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:16:59.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:16:59.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:00.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:00.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:00.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:00.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:00.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:00.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:00.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:00.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:00.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:00.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:00.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:00.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:00.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:00.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:00.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:00.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:00.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:00.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:00.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:00.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:01.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:01.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:01.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:01.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:01.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:01.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:01.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:01.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:01.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:01.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:01.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:01.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:01.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:01.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:01.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:01.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:01.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:01.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:01.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:01.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:02.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:02.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:02.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:02.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:02.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:02.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:02.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:02.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:02.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:02.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:02.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:02.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:02.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:02.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:02.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:02.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:02.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:02.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:02.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:02.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:03.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:03.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:03.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:03.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:03.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:03.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:03.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:03.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:03.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:03.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:03.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:03.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:03.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:03.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:03.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:03.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:03.996 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:03.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:03.997 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:03.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:04.970 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:04.929+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 995 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:04.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:04.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:04.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:04.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:04.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:04.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:04.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:04.703+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:04.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:04.703+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:04.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:04.703+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:04.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:04.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:04.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:04.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:04.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:04.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:04.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:04.804+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:05.621 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-02-19T19:17:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:05.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:05.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:05.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:05.812+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:05.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:05.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:05.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:05.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:05.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:05.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:05.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:05.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:05.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:05.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:05.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:05.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:06.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:06.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:06.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:06.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:06.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:06.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:06.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:06.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:06.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:06.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:06.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:06.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:06.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:06.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:06.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:06.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:06.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:06.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:06.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:06.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:07.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:07.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:07.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:07.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:07.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:07.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:07.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:07.855+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:07.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:07.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:07.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:07.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:07.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:07.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:07.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:07.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:07.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:07.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:07.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:07.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:08.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:08.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:08.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:08.843+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:08.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:08.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:08.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:08.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:08.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:08.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:08.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:08.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:08.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:08.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:08.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:08.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:09.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:09.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:09.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:09.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:09.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:09.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:09.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:09.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:09.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:09.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:09.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:09.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:09.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:09.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:09.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:09.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:09.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:09.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:09.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:09.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:10.210 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:09.931+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1031 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:10.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:10.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:10.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:10.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:10.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:10.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:10.904 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:10.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:10.904 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:10.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:10.905 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:10.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:11.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:10.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:11.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:10.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:11.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:10.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:11.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:10.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:11.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:11.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:11.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:11.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:11.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:11.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:11.895 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:11.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:11.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:11.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:11.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:11.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:12.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:11.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:12.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:11.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:12.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:11.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:12.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:11.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:12.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:12.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:12.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:12.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:12.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:12.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:12.871 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:12.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:12.871 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:12.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:12.871 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:12.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:13.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:12.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:13.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:12.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:13.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:12.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:13.245 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:12.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:13.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:13.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:13.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:13.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:13.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:13.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:13.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:13.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:13.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:13.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:13.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:13.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:14.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:13.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:13.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:13.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:13.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:14.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:14.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:14.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:14.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:14.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:14.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:14.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:14.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:14.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:14.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:14.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:14.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:14.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:14.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:14.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:14.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:14.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:14.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:14.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:14.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:15.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:17:15.156+0000 7f28ba0b8700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:17:15.174 INFO:teuthology.orchestra.run.smithi039.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:17:15.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:14.931+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1074 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:15.848 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:15.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:15.848 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:15.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:15.848 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:15.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:15.848 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:15.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:15.848 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:15.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:15.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:15.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:15.929 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps' 2024-02-19T19:17:16.149 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/mon.smithi039/config 2024-02-19T19:17:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:15.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:15.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:15.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:15.831+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:16.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:16.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:16.806 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:16.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:16.806 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:16.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:17.243 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:16.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:17.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:16.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:17.244 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:16.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:17.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:16.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:17.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:16.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:17.245 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:16.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:17.245 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:16.811+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:17.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:17.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:17.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:17.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:17.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:17.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:17.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:17.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:17.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:17.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:17.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:17.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:17.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:17.822+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:17.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:17.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:17.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:17.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:17.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:17.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:18.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:18.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:18.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:18.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:18.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:18.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:18.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:18.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:18.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:18.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:18.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:18.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:18.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:18.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:18.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:18.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:18.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:18.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:18.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:18.831+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:19.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:19.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:19.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:19.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:19.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:19.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:20.241 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:19.932+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1156 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:20.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:19.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:20.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:19.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:20.242 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:19.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:20.242 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:19.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:20.242 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:19.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:20.242 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:19.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:20.242 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:19.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:20.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:20.605+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:20.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:20.605+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:20.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:20.605+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:20.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:20.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:20.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:20.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:20.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:20.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:20.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:20.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:20.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:20.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:20.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:20.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:20.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:20.837+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:21.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:21.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:21.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:21.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:21.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:21.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:21.847 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:21.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:21.847 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:21.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:21.847 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:21.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:22.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:21.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:22.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:21.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:22.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:21.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:22.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:21.830+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:22.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:22.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:22.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:22.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:22.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:22.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:22.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:22.788+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:22.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:22.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:22.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:22.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:22.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:22.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:22.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:22.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:22.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:22.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:22.996 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:22.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:23.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:23.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:23.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:23.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:23.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:23.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:23.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:23.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:23.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:23.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:23.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:23.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:24.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:23.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:24.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:23.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:24.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:23.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:24.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:23.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:24.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:24.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:24.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:24.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:24.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:24.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:24.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:24.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:24.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:24.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:24.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:24.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:24.826+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:25.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:24.933+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1239 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:25.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:25.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:25.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:25.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:25.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:25.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:25.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:25.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:25.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:25.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:25.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:25.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:25.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:25.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:25.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:25.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:25.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:25.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:25.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:25.787+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:26.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:26.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:26.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:26.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:26.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:26.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:26.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:26.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:26.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:26.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:26.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:26.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:26.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:26.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:26.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:26.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:26.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:26.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:26.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:26.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:27.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:27.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:27.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:27.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:27.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:27.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:27.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:27.788+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:27.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:27.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:27.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:27.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:27.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:27.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:27.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:27.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:27.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:27.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:27.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:27.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:28.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:28.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:28.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:28.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:28.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:28.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:28.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:28.741+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:28.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:28.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:28.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:28.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:28.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:28.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:28.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:28.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:28.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:28.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:28.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:28.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:29.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:29.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:29.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:29.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:29.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:29.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:29.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:29.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:29.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:29.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:29.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:29.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:29.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:29.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:29.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:29.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:29.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:29.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:29.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:29.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:30.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:29.933+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1321 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:30.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:30.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:30.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:30.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:30.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:30.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:30.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:30.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:30.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:30.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:30.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:30.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:30.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:30.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:30.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:30.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:30.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:30.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:30.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:30.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:31.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:31.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:31.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:31.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:31.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:31.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:31.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:31.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:31.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:31.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:31.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:31.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:31.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:31.848+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:31.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:31.848+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:31.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:31.848+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:31.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:31.848+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:32.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:32.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:32.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:32.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:32.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:32.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:32.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:32.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:32.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:32.873+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:32.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:32.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:32.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:32.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:33.853 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:33.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:33.853 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:33.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:33.853 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:33.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:33.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:33.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:33.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:33.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:33.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:33.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:34.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:33.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:34.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:33.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:34.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:33.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:34.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:33.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:34.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:34.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:34.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:34.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:34.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:34.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:34.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:34.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:34.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:34.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:34.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:34.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:34.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:34.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:34.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:34.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:34.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:34.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:34.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:34.798+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:35.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:34.934+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1404 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:35.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:35.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:35.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:35.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:35.743 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:35.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:36.243 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:35.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:36.243 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:35.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:36.243 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:35.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:36.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:35.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:36.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:35.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:36.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:35.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:36.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:35.818+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:36.826 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:36.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:36.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:36.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:36.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:36.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:36.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:36.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:36.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:36.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:36.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:36.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:37.242 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:36.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:37.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:36.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:37.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:36.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:37.243 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:36.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:37.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:37.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:37.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:37.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:37.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:37.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:37.713+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:37.713+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:37.713+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:37.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:37.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:37.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:37.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:37.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:38.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:38.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:38.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:38.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:38.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:38.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:38.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:38.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:38.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:38.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:38.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:38.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:38.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:38.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:38.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:38.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:38.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:38.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:38.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:38.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:39.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:39.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:39.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:39.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:39.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:39.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:39.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:39.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:39.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:39.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:39.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:39.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:39.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:39.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:39.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:39.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:39.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:39.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:39.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:39.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:40.243 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:39.934+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1486 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:40.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:40.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:40.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:40.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:40.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:40.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:40.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:40.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:40.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:40.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:40.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:40.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:40.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:40.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:40.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:40.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:40.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:40.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:40.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:40.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:41.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:41.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:41.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:41.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:41.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:41.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:41.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:41.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:41.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:41.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:41.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:41.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:41.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:41.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:41.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:41.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:41.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:41.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:41.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:41.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:42.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:42.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:42.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:42.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:42.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:42.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:42.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:42.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:42.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:42.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:42.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:42.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:42.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:42.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:42.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:42.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:42.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:42.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:42.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:42.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:43.874 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:43.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:43.875 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:43.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:43.875 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:43.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:43.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:43.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:43.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:43.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:43.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:43.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:43.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:43.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:43.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:43.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:43.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:43.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:43.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:43.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:44.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:44.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:44.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:44.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:44.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:44.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:44.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:44.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:44.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:44.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:44.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:44.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:44.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:44.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:44.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:44.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:44.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:44.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:44.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:44.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:45.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:44.935+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1569 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:45.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:45.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:45.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:45.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:45.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:45.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:45.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:45.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:45.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:45.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:45.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:45.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:45.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:45.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:45.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:45.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:45.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:45.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:45.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:45.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:46.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:46.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:46.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:46.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:46.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:46.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:46.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:46.659+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:46.961 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:46.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:46.961 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:46.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:46.961 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:46.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:46.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:46.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:46.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:46.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:46.962 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:46.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:47.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:47.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:47.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:47.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:47.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:47.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:47.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:47.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:47.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:47.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:47.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:47.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:47.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:47.708+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:47.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:47.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:47.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:47.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:47.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:47.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:48.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:48.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:48.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:48.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:48.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:48.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:48.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:48.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:48.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:48.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:48.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:48.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:48.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:48.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:48.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:48.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:48.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:48.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:48.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:49.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:49.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:49.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:49.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:49.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:49.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:49.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:49.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:49.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:49.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:49.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:49.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:49.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:49.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:49.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:49.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:49.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:49.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:49.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:49.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:50.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:49.936+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1651 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:50.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:50.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:50.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:50.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:50.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:50.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:50.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:50.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:50.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:50.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:50.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:50.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:50.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:50.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:50.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:50.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:50.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:50.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:50.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:50.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:51.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:51.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:51.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:51.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:51.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:51.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:51.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:51.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:51.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:51.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:51.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:51.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:51.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:51.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:51.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:51.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:51.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:51.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:51.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:51.666+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:52.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:52.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:52.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:52.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:52.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:52.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:52.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:52.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:52.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:52.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:52.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:52.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:52.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:52.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:52.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:53.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:53.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:53.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:53.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:53.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:53.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:53.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:53.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:53.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:53.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:53.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:53.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:53.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:53.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:53.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:53.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:53.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:53.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:53.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:53.706+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:54.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:54.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:54.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:54.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:54.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:54.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:54.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:54.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:54.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:54.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:54.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:54.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:54.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:54.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:54.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:54.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:54.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:54.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:54.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:54.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:55.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:54.937+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1734 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:17:55.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:55.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:55.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:55.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:55.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:55.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:55.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:55.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:55.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:55.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:55.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:55.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:55.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:55.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:55.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:55.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:55.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:55.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:55.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:55.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:56.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:56.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:56.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:56.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:56.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:56.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:56.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:56.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:56.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:56.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:56.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:56.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:56.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:56.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:56.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:56.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:56.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:56.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:56.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:56.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:57.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:57.715+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:57.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:57.715+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:57.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:57.715+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:57.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:57.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:57.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:57.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:57.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:57.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:57.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:57.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:57.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:57.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:57.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:57.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:57.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:57.719+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:58.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:58.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:58.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:58.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:58.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:58.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:58.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:58.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:58.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:58.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:58.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:58.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:58.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:58.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:58.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:58.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:58.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:58.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:58.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:58.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:59.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:59.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:59.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:59.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:59.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:17:59.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:17:59.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:59.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:59.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:59.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:59.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:17:59.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:17:59.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:59.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:59.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:59.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:59.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:59.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:17:59.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:17:59.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:00.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:17:59.939+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1816 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:00.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:00.717+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:00.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:00.717+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:00.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:00.717+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:00.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:00.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:00.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:00.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:00.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:00.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:00.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:00.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:00.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:01.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:01.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:01.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:01.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:01.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:01.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:01.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:01.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:01.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:01.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:01.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:01.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:01.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:01.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:01.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:01.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:01.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:01.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:01.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:01.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:02.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:02.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:02.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:02.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:02.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:02.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:02.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:02.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:02.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:02.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:02.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:02.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:02.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:02.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:02.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:02.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:02.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:02.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:02.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:02.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:03.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:03.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:03.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:03.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:03.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:03.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:03.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:03.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:03.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:03.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:03.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:03.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:03.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:03.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:03.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:03.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:03.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:03.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:03.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:03.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:04.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:04.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:04.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:04.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:04.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:04.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:04.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:04.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:04.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:04.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:04.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:04.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:04.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:04.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:04.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:04.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:04.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:04.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:04.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:04.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:05.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:04.940+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1899 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:05.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:05.703+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:05.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:05.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:05.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:05.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:05.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:05.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:05.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:05.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:05.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:05.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:05.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:05.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:05.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:05.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:05.694+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:06.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:06.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:06.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:06.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:06.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:06.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:06.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:06.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:06.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:06.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:06.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:06.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:06.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:06.675+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:06.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:06.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:06.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:06.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:06.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:06.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:07.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:07.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:07.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:07.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:07.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:07.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:07.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:07.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:07.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:07.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:07.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:07.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:07.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:07.670+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:07.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:07.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:07.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:07.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:07.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:07.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:08.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:08.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:08.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:08.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:08.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:08.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:08.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:08.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:08.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:08.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:08.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:08.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:08.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:08.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:08.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:08.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:08.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:08.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:09.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:09.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:09.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:09.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:09.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:09.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:09.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:09.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:09.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:09.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:09.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:09.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:09.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:09.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:09.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:09.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:09.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:09.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:09.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:09.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:10.242 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:09.941+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 1981 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:10.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:10.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:10.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:10.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:10.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:10.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:10.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:10.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:10.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:10.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:10.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:10.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:10.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:10.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:10.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:10.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:10.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:10.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:10.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:10.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:11.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:11.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:11.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:11.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:11.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:11.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:11.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:11.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:11.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:11.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:11.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:11.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:11.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:11.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:11.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:11.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:11.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:11.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:11.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:11.603+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:12.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:12.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:12.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:12.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:12.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:12.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:12.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:12.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:12.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:12.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:12.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:12.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:12.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:12.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:12.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:12.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:12.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:12.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:12.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:12.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:13.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:13.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:13.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:13.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:13.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:13.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:13.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:13.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:13.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:13.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:13.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:13.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:13.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:13.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:13.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:13.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:13.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:13.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:13.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:13.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:14.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:14.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:14.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:14.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:14.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:14.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:14.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:14.623+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:14.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:14.623+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:14.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:14.623+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:14.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:14.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:14.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:14.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:14.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:14.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:14.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:14.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:15.241 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:14.941+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2063 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:15.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:15.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:15.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:15.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:15.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:15.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:15.931 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:15.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:15.931 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:15.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:15.931 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:15.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:15.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:15.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:15.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:15.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:15.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:15.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:15.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:15.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:16.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:16.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:16.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:16.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:16.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:16.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:16.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:16.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:16.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:16.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:16.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:16.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:16.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:16.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:16.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:16.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:16.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:16.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:16.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:16.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:17.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:17.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:17.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:17.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:17.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:17.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:17.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:17.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:17.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:17.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:17.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:17.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:17.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:17.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:17.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:17.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:17.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:17.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:17.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:17.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:18.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:18.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:18.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:18.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:18.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:18.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:18.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:18.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:18.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:18.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:18.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:18.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:18.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:18.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:18.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:18.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:18.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:18.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:18.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:18.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:19.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:19.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:19.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:19.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:19.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:19.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:19.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:19.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:19.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:19.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:19.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:19.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:19.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:19.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:19.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:19.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:19.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:19.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:19.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:19.670+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:20.241 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:19.941+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2146 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:20.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:20.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:20.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:20.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:20.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:20.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:20.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:20.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:20.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:20.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:20.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:20.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:20.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:20.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:20.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:20.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:20.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:20.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:20.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:20.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:21.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:21.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:21.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:21.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:21.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:21.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:21.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:21.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:21.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:21.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:21.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:21.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:21.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:21.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:21.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:21.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:21.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:22.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:22.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:22.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:22.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:22.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:22.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:22.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:22.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:22.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:22.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:22.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:22.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:22.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:22.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:22.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:22.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:22.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:22.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:22.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:22.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:23.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:23.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:23.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:23.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:23.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:23.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:23.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:23.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:23.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:23.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:23.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:23.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:23.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:23.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:24.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:23.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:24.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:23.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:24.242 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:23.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:24.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:24.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:24.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:24.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:24.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:24.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:24.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:24.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:24.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:24.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:24.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:24.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:24.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:24.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:24.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:24.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:24.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:24.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:24.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:24.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:25.241 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:24.942+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2228 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:25.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:25.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:25.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:25.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:25.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:25.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:25.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:25.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:25.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:25.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:25.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:25.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:25.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:25.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:25.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:25.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:25.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:25.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:25.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:25.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:26.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:26.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:26.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:26.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:26.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:26.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:26.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:26.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:26.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:26.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:26.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:26.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:26.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:26.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:26.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:26.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:26.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:26.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:26.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:26.627+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:27.864 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:27.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:27.864 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:27.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:27.864 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:27.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:27.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:27.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:27.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:27.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:27.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:27.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:27.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:27.636+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:28.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:27.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:28.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:27.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:28.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:27.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:28.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:28.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:28.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:28.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:28.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:28.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:28.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:28.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:28.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:28.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:28.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:28.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:28.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:28.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:29.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:28.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:29.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:28.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:29.242 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:28.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:29.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:29.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:29.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:29.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:29.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:29.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:29.837 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:29.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:29.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:29.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:29.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:29.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:29.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:29.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:30.087 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:29.943+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2311 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:30.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:29.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:30.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:29.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:30.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:29.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:30.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:30.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:30.833 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:30.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:30.833 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:30.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:30.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:30.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:30.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:30.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:30.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:30.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:30.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:30.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:31.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:30.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:31.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:30.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:31.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:30.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:31.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:31.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:31.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:31.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:31.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:31.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:31.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:31.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:31.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:31.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:31.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:31.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:31.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:31.592+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:32.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:31.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:32.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:31.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:32.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:31.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:32.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:32.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:32.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:32.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:32.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:32.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:32.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:32.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:32.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:32.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:32.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:32.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:32.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:32.631+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:33.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:32.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:33.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:32.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:33.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:32.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:33.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:33.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:33.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:33.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:33.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:33.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:33.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:33.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:33.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:33.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:33.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:33.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:33.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:33.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:34.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:33.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:34.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:33.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:34.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:33.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:34.960 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:34.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:34.960 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:34.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:34.960 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:34.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:34.961 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:34.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:34.961 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:34.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:34.961 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:34.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:34.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:34.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:34.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:34.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:34.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:34.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:34.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:34.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:35.241 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:34.944+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2393 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:35.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:35.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:35.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:35.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:35.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:35.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:35.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:35.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:35.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:35.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:35.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:35.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:35.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:35.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:35.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:35.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:35.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:35.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:35.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:35.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:36.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:36.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:36.804 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:36.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:36.804 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:36.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:36.804 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:36.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:36.804 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:36.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:36.804 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:36.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:36.804 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:36.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:37.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:36.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:37.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:36.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:37.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:36.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:37.802 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:37.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:37.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:37.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:37.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:37.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:37.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:37.730+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:37.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:37.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:37.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:37.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:37.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:37.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:38.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:37.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:38.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:37.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:38.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:37.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:38.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:38.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:38.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:38.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:38.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:38.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:38.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:38.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:38.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:38.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:38.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:38.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:38.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:38.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:39.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:38.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:39.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:38.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:39.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:38.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:39.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:39.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:39.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:39.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:39.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:39.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:39.991 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:39.945+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2476 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:39.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:39.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:39.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:39.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:39.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:39.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:39.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:39.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:39.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:39.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:39.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:39.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:39.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:39.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:40.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:40.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:40.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:40.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:40.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:40.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:40.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:40.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:40.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:40.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:40.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:40.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:40.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:40.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:41.209 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:40.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:41.209 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:40.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:41.209 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:40.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:41.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:41.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:41.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:41.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:41.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:41.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:41.837 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:41.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:41.837 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:41.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:41.837 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:41.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:41.837 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:41.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:42.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:41.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:42.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:41.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:42.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:41.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:42.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:42.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:42.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:42.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:42.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:42.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:42.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:42.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:42.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:42.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:42.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:42.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:42.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:42.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:43.240 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:42.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:43.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:42.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:43.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:42.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:43.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:43.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:43.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:43.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:43.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:43.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:43.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:43.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:43.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:43.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:43.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:43.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:43.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:43.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:44.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:43.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:44.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:43.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:44.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:43.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:44.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:44.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:44.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:44.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:44.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:44.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:44.741 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:44.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:44.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:44.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:44.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:44.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:44.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:44.609+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:45.085 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:44.946+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2558 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:45.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:44.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:45.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:44.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:45.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:44.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:45.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:45.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:45.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:45.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:45.782 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:45.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:45.783 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:45.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:45.783 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:45.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:46.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:45.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:46.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:45.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:46.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:45.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:46.826 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:46.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:46.826 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:46.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:46.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:46.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:46.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:46.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:46.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:46.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:46.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:46.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:46.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:46.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:46.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:46.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:46.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:46.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:46.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:46.602+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:47.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:47.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:47.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:47.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:47.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:47.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:47.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:47.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:47.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:47.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:47.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:47.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:47.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:47.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:47.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:47.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:47.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:47.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:47.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:47.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:48.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:48.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:48.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:48.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:48.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:48.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:48.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:48.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:48.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:48.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:48.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:48.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:48.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:48.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:48.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:48.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:48.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:49.962 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:49.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:49.963 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:49.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:49.963 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:49.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:49.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:49.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:49.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:49.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:49.964 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:49.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:49.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:49.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:49.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:49.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:49.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:49.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:49.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:49.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:50.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:49.948+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2641 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:50.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:50.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:50.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:50.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:50.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:50.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:50.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:50.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:50.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:50.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:50.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:50.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:50.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:50.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:50.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:50.742 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:50.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:51.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:51.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:51.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:51.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:51.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:51.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:51.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:51.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:51.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:51.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:51.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:51.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:51.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:51.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:51.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:51.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:51.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:51.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:51.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:51.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:52.990 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:52.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:52.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:52.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:52.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:52.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:52.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:52.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:52.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:52.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:52.992 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:52.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:52.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:52.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:52.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:52.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:52.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:52.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:52.992 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:52.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:53.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:53.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:53.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:53.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:53.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:53.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:53.797 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:53.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:53.797 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:53.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:53.797 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:53.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:54.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:53.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:54.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:53.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:54.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:53.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:54.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:53.782+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:54.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:54.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:54.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:54.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:54.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:54.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:54.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:54.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:54.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:54.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:54.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:54.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:55.090 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:54.949+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2723 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:18:55.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:54.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:55.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:54.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:55.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:54.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:55.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:54.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:55.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:55.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:55.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:55.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:55.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:55.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:55.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:55.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:55.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:55.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:55.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:55.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:56.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:55.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:56.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:55.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:56.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:55.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:56.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:55.772+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:56.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:56.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:56.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:56.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:56.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:56.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:56.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:56.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:56.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:56.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:56.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:56.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:56.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:56.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:56.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:56.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:56.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:56.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:56.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:56.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:57.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:57.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:57.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:57.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:57.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:57.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:57.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:57.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:57.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:57.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:57.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:57.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:58.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:57.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:58.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:57.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:58.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:57.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:58.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:57.841+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:58.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:58.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:58.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:58.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:58.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:58.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:58.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:58.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:58.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:58.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:58.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:58.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:59.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:58.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:59.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:58.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:59.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:58.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:18:59.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:58.865+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:59.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:59.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:59.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:59.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:59.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:18:59.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:18:59.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:59.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:59.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:59.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:18:59.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:18:59.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:00.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:18:59.950+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2806 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:00.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:59.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:00.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:59.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:00.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:59.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:00.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:18:59.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:00.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:00.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:00.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:00.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:00.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:00.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:00.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:00.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:00.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:00.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:00.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:00.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:01.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:00.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:01.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:00.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:01.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:00.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:01.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:00.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:01.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:01.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:01.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:01.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:01.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:01.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:01.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:01.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:01.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:01.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:01.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:01.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:02.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:01.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:02.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:01.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:02.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:01.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:02.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:01.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:02.658 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:02.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:02.658 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:02.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:02.658 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:02.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:02.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:02.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:02.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:02.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:02.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:02.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:03.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:02.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:03.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:02.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:03.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:02.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:03.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:02.941+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:03.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:03.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:03.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:03.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:03.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:03.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:03.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:03.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:03.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:03.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:03.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:03.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:04.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:03.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:04.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:03.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:04.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:03.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:04.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:03.952+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:04.627 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:04.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:04.627 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:04.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:04.627 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:04.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:04.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:04.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:04.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:04.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:04.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:04.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:05.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:04.951+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2888 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:05.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:04.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:05.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:04.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:05.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:04.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:05.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:04.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:05.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:05.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:05.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:05.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:05.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:05.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:05.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:05.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:05.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:05.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:05.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:05.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:06.369 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:06.370 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:06.370 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:06.370 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:06.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:06.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:06.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:06.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:06.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:06.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:06.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:06.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:06.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:06.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:06.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:06.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:07.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:07.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:07.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:07.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:06.975+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:07.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:07.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:07.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:07.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:07.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:07.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:07.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:07.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:07.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:07.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:07.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:07.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:08.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:08.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:08.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:08.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:08.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:08.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:08.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:08.019+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:08.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:08.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:08.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:08.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:08.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:08.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:08.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:08.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:08.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:08.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:08.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:08.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:09.421 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:09.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:09.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:09.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:09.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:09.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:09.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:09.047+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:09.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:09.604+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:09.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:09.604+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:09.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:09.604+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:09.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:09.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:09.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:09.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:09.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:09.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:10.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:09.951+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 2971 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:10.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:10.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:10.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:10.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:10.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:10.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:10.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:10.078+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:10.657 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:10.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:10.657 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:10.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:10.657 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:10.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:10.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:10.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:10.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:10.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:10.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:10.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:11.356 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:11.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:11.356 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:11.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:11.356 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:11.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:11.357 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:11.031+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:11.661 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:11.342+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:11.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:11.342+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:11.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:11.342+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:11.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:11.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:11.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:11.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:11.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:11.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:12.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:12.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:12.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:12.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:12.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:12.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:12.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:12.015+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:12.645 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:12.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:12.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:12.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:12.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:12.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:12.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:12.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:12.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:12.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:12.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:12.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:13.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:13.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:13.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:13.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:13.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:13.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:13.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:13.049+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:13.672 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:13.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:13.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:13.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:13.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:13.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:13.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:13.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:13.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:13.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:13.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:13.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:14.381 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:14.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:14.381 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:14.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:14.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:14.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:14.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:14.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:14.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:14.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:14.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:14.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:14.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:14.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:14.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:14.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:14.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:14.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:14.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:14.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:15.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:14.953+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3053 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:15.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:15.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:15.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:15.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:15.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:15.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:15.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:15.015+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:15.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:15.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:15.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:15.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:15.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:15.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:15.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:15.618+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:15.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:15.618+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:15.742 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:15.618+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:16.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:16.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:16.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:16.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:16.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:16.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:16.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:16.001+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:16.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:16.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:16.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:16.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:16.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:16.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:16.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:16.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:16.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:16.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:16.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:16.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:17.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:17.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:17.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:17.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:17.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:17.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:17.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:17.047+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:17.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:17.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:17.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:17.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:17.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:17.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:17.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:17.567+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:17.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:17.567+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:17.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:17.567+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:18.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:18.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:18.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:18.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:18.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:18.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:18.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:18.084+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:18.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:18.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:18.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:18.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:18.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:18.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:18.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:18.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:18.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:18.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:18.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:18.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:19.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:19.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:19.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:19.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:19.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:19.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:19.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:19.037+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:19.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:19.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:19.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:19.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:19.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:19.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:19.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:19.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:19.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:19.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:19.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:19.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:20.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:19.954+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3136 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:20.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:20.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:20.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:20.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:20.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:20.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:20.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:20.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:20.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:20.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:20.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:20.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:20.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:20.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:20.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:20.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:21.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:21.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:21.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:21.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:20.964+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:21.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:21.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:21.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:21.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:21.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:21.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:21.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:21.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:21.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:21.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:21.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:21.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:22.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:21.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:22.269 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:21.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:22.269 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:21.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:22.269 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:21.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:22.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:22.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:22.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:22.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:22.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:22.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:22.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:22.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:22.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:22.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:22.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:22.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:23.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:22.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:23.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:22.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:23.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:22.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:23.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:22.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:23.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:23.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:23.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:23.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:23.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:23.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:23.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:23.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:23.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:23.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:23.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:23.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:24.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:23.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:24.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:23.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:24.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:23.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:24.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:23.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:24.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:24.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:24.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:24.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:24.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:24.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:24.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:24.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:24.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:24.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:24.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:24.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:25.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:24.955+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3218 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:25.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:25.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:25.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:25.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:25.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:25.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:25.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:25.001+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:25.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:25.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:25.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:25.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:25.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:25.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:25.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:25.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:25.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:25.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:25.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:25.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:26.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:26.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:26.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:26.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:26.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:26.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:26.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:26.021+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:26.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:26.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:26.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:26.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:26.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:26.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:26.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:26.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:26.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:26.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:26.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:26.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:27.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:27.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:27.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:27.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:27.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:27.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:27.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:27.052+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:27.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:27.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:27.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:27.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:27.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:27.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:27.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:27.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:27.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:27.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:27.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:27.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:28.421 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:28.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:28.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:28.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:28.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:28.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:28.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:28.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:28.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:28.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:28.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:28.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:28.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:28.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:28.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:28.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:28.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:28.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:28.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:28.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:29.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:29.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:29.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:29.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:29.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:29.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:29.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:29.095+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:29.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:29.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:29.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:29.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:29.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:29.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:29.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:29.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:29.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:29.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:29.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:29.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:30.240 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:29.956+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3301 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:30.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:30.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:30.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:30.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:30.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:30.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:30.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:30.071+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:30.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:30.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:30.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:30.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:30.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:30.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:30.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:30.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:30.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:30.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:30.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:30.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:31.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:31.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:31.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:31.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:31.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:31.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:31.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:31.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:31.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:31.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:31.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:31.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:31.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:31.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:31.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:31.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:31.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:31.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:31.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:31.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:32.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:32.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:32.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:32.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:32.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:32.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:32.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:32.051+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:32.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:32.416+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:32.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:32.416+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:32.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:32.416+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:32.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:32.550+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:32.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:32.550+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:32.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:32.550+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:33.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:33.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:33.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:33.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:33.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:33.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:33.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:33.070+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:33.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:33.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:33.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:33.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:33.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:33.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:33.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:33.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:33.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:33.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:33.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:33.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:34.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:34.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:34.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:34.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:34.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:34.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:34.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:34.049+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:34.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:34.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:34.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:34.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:34.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:34.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:34.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:34.569+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:34.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:34.569+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:34.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:34.569+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:35.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:34.957+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3383 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:35.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:35.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:35.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:35.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:35.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:35.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:35.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:35.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:35.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:35.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:35.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:35.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:35.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:35.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:35.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:35.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:35.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:35.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:35.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:35.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:36.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:36.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:36.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:36.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:36.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:36.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:36.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:36.001+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:36.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:36.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:36.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:36.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:36.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:36.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:36.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:36.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:36.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:36.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:36.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:36.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:37.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:37.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:37.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:37.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:37.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:37.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:37.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:37.006+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:37.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:37.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:37.990 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:37.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:37.990 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:37.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:37.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:37.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:37.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:37.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:37.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:37.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:38.489 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.007+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:38.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.007+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:38.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.007+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:38.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.007+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:38.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:38.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:38.990 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:38.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:38.990 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:38.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:38.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:38.496+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:38.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:38.496+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:38.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:38.496+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:39.489 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:39.489 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:39.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:39.490 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:38.984+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:39.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:39.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:39.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:39.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:39.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:39.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:39.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:39.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:39.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:39.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:39.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:39.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:40.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:39.959+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3466 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:40.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:39.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:40.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:39.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:40.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:39.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:40.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:39.994+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:40.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:40.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:40.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:40.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:40.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:40.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:40.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:40.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:40.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:40.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:40.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:40.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:41.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:40.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:41.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:40.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:41.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:40.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:41.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:40.985+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:41.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:41.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:41.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:41.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:41.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:41.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:41.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:41.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:41.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:41.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:41.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:41.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:42.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:41.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:42.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:41.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:42.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:41.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:42.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:41.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:42.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:42.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:42.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:42.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:42.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:42.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:42.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:42.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:42.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:42.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:42.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:42.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:43.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:42.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:43.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:42.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:43.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:42.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:43.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:42.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:43.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:43.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:43.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:43.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:43.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:43.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:43.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:43.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:43.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:43.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:43.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:43.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:44.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:43.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:44.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:43.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:44.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:43.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:44.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:43.985+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:44.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:44.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:44.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:44.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:44.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:44.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:44.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:44.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:44.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:44.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:44.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:44.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:45.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:44.960+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3548 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:45.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:44.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:45.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:44.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:45.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:44.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:45.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:44.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:45.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:45.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:45.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:45.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:45.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:45.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:45.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:45.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:45.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:45.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:45.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:45.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:46.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:45.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:46.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:45.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:46.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:45.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:46.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:45.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:46.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:46.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:46.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:46.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:46.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:46.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:46.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:46.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:46.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:46.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:46.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:46.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:47.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:46.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:47.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:46.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:47.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:46.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:47.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:46.938+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:47.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:47.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:47.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:47.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:47.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:47.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:47.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:47.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:47.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:47.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:47.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:47.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:48.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:47.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:48.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:47.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:48.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:47.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:48.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:47.899+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:48.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:48.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:48.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:48.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:48.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:48.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:48.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:48.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:48.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:48.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:48.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:48.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:49.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:48.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:49.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:48.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:49.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:48.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:49.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:48.865+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:49.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:49.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:49.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:49.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:49.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:49.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:49.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:49.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:49.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:49.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:49.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:49.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:50.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:49.961+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3635 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:50.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:49.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:50.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:49.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:50.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:49.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:50.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:49.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:50.846 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:50.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:50.847 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:50.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:50.847 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:50.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:50.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:50.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:50.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:50.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:50.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:50.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:51.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:50.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:51.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:50.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:51.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:50.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:51.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:50.834+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:51.852 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:51.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:51.853 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:51.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:51.853 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:51.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:51.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:51.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:51.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:51.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:51.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:51.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:52.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:51.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:52.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:51.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:52.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:51.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:52.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:51.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:52.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:52.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:52.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:52.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:52.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:52.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:52.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:52.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:52.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:52.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:52.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:52.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:53.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:52.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:53.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:52.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:53.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:52.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:53.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:52.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:53.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:53.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:53.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:53.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:53.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:53.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:53.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:53.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:53.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:53.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:53.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:53.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:54.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:53.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:54.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:53.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:54.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:53.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:54.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:53.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:54.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:54.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:54.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:54.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:54.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:54.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:54.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:54.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:54.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:54.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:54.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:54.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:55.097 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:54.963+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3721 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:19:55.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:54.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:55.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:54.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:55.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:54.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:55.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:54.800+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:55.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:55.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:55.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:55.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:55.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:55.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:55.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:55.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:55.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:55.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:55.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:55.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:56.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:55.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:56.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:55.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:56.160 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:55.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:56.160 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:55.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:56.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:56.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:56.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:56.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:56.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:56.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:56.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:56.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:56.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:56.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:56.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:56.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:57.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:56.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:57.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:56.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:57.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:56.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:57.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:56.804+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:57.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:57.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:57.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:57.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:57.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:57.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:57.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:57.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:57.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:57.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:57.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:57.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:58.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:57.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:58.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:57.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:58.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:57.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:58.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:57.839+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:58.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:58.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:58.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:58.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:58.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:58.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:58.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:58.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:58.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:58.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:58.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:58.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:59.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:58.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:59.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:58.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:59.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:58.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:19:59.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:58.881+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:59.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:59.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:59.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:59.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:59.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:19:59.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:19:59.910 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:59.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:59.910 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:59.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:19:59.911 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:19:59.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:00.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:19:59.963+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3824 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:00.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:59.897+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:00.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:59.897+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:00.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:59.897+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:00.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:19:59.897+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:00.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:00.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:00.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:00.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:00.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:00.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:00.960 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:00.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:00.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:00.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:00.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:00.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:01.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:00.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:01.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:00.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:01.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:00.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:01.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:00.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:01.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:01.614+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:01.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:01.614+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:01.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:01.614+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:01.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:01.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:01.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:01.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:01.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:01.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:02.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:01.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:02.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:01.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:02.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:01.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:02.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:01.928+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:02.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:02.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:02.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:02.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:02.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:02.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:02.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:02.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:02.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:02.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:02.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:02.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:03.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:02.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:03.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:02.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:03.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:02.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:03.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:02.916+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:03.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:03.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:03.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:03.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:03.740 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:03.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:03.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:03.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:03.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:03.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:03.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:03.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:04.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:03.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:04.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:03.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:04.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:03.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:04.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:03.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:04.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:04.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:04.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:04.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:04.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:04.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:04.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:04.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:04.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:04.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:04.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:04.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:05.239 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:04.964+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3906 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:05.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:04.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:05.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:04.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:05.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:04.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:05.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:04.915+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:05.667 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:05.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:05.667 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:05.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:05.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:05.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:05.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:05.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:05.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:05.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:05.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:05.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:06.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:05.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:06.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:05.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:06.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:05.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:06.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:05.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:06.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:06.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:06.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:06.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:06.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:06.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:06.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:06.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:06.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:06.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:06.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:06.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:06.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:06.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:06.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:06.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:06.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:06.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:06.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:06.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:07.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:07.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:07.489 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:07.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:07.489 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:07.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:08.239 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:07.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:08.239 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:07.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:08.239 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:07.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:08.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:07.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:08.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:07.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:08.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:07.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:08.240 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:07.928+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:08.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:08.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:08.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:08.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:08.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:08.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:08.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:08.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:08.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:08.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:08.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:08.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:08.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:08.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:08.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:08.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:08.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:08.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:08.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:08.938+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:09.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:09.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:09.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:09.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:09.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:09.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:09.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:09.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:09.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:09.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:09.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:09.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:10.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:09.965+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 3989 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:10.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:10.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:10.708 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:10.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:10.708 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:10.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:10.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:10.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:10.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:10.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:10.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:10.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:10.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:10.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:10.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:10.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:10.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:10.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:10.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:10.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:11.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:11.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:11.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:11.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:11.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:11.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:11.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:11.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:11.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:11.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:11.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:11.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:12.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:11.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:12.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:11.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:12.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:11.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:12.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:11.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:12.488 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:12.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:12.489 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:12.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:12.489 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:12.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:12.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:12.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:12.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:12.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:12.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:12.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:12.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:12.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:12.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:12.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:12.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:12.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:12.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:12.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:13.679 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:13.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:13.679 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:13.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:13.679 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:13.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:13.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:13.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:13.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:13.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:13.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:13.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:13.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:13.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:13.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:13.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:13.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:13.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:13.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:13.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:14.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:14.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:14.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:14.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:14.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:14.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:14.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:14.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:14.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:14.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:14.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:14.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:15.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:14.966+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4074 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:15.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:14.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:15.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:14.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:15.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:14.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:15.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:14.923+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:15.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:15.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:15.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:15.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:15.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:15.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:15.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:15.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:15.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:15.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:15.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:15.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:16.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:15.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:16.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:15.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:16.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:15.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:16.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:15.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:16.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:16.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:16.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:16.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:16.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:16.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:16.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:16.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:16.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:16.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:16.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:16.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:17.270 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:16.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:17.270 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:16.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:17.271 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:16.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:17.271 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:16.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:17.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:17.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:17.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:17.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:17.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:17.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:17.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:17.629+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:17.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:17.629+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:17.989 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:17.629+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:18.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:17.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:18.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:17.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:18.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:17.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:18.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:17.980+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:18.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:18.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:18.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:18.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:18.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:18.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:18.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:18.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:18.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:18.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:18.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:18.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:19.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:18.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:19.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:18.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:19.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:18.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:19.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:18.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:19.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:19.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:19.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:19.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:19.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:19.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:19.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:19.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:19.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:19.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:19.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:19.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:20.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:19.967+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4162 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:20.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:19.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:20.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:19.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:20.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:19.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:20.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:19.968+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:20.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:20.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:20.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:20.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:20.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:20.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:20.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:20.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:20.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:20.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:20.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:20.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:21.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:20.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:21.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:20.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:21.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:20.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:21.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:20.996+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:21.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:21.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:21.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:21.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:21.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:21.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:21.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:21.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:21.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:21.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:21.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:21.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:22.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:21.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:22.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:21.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:22.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:21.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:22.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:21.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:22.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:22.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:22.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:22.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:22.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:22.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:22.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:22.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:22.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:22.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:22.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:22.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:23.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:22.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:23.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:22.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:23.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:22.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:23.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:22.972+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:23.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:23.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:23.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:23.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:23.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:23.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:23.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:23.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:23.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:23.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:23.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:23.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:24.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:24.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:24.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:24.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:24.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:24.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:24.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:24.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:24.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:24.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:24.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:24.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:24.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:25.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:24.969+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4244 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:25.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:25.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:25.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:25.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:24.968+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:25.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:25.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:25.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:25.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:25.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:25.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:25.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:25.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:25.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:25.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:25.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:25.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:26.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:25.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:26.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:25.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:26.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:25.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:26.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:25.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:26.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:26.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:26.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:26.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:26.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:26.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:26.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:26.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:26.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:26.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:26.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:26.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:27.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:26.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:27.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:26.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:27.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:26.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:27.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:26.964+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:27.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:27.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:27.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:27.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:27.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:27.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:27.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:27.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:27.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:28.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:27.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:28.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:27.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:28.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:27.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:28.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:27.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:28.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:28.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:28.787 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:28.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:28.787 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:28.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:28.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:28.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:28.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:28.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:28.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:28.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:29.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:28.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:29.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:28.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:29.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:28.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:29.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:28.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:29.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:29.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:29.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:29.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:29.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:29.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:29.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:29.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:29.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:29.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:29.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:29.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:30.103 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:29.971+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4332 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:30.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:30.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:30.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:30.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:30.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:30.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:30.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:30.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:30.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:30.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:30.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:30.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:30.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:30.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:30.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:30.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:31.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:31.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:31.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:31.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:30.982+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:31.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:31.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:31.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:31.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:31.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:31.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:31.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:31.448+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:31.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:31.448+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:31.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:31.448+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:32.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:32.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:32.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:32.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.010+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:32.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:32.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:32.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:32.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:32.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:32.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:32.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:32.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:32.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:32.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:32.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:32.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:33.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:33.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:33.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:33.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:32.977+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:33.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:33.599+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:33.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:33.599+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:33.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:33.599+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:33.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:33.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:33.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:33.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:33.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:33.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:34.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:33.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:34.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:33.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:34.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:33.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:34.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:33.957+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:34.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:34.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:34.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:34.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:34.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:34.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:34.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:34.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:34.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:34.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:34.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:34.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:35.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:34.972+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4424 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:35.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:34.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:35.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:34.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:35.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:34.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:35.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:34.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:35.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:35.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:35.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:35.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:35.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:35.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:35.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:35.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:35.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:35.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:35.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:35.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:36.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:35.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:36.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:35.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:36.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:35.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:36.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:35.921+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:36.556 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:36.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:36.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:36.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:36.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:36.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:36.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:36.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:36.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:36.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:36.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:36.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:37.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:36.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:37.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:36.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:37.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:36.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:37.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:36.903+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:37.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:37.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:37.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:37.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:37.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:37.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:37.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:37.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:37.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:37.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:37.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:37.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:38.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:37.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:38.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:37.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:38.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:37.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:38.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:37.944+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:38.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:38.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:38.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:38.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:38.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:38.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:38.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:38.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:38.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:38.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:38.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:38.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:39.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:38.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:39.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:38.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:39.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:38.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:39.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:38.925+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:39.902 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:39.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:39.902 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:39.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:39.903 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:39.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:39.903 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:39.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:39.903 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:39.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:39.903 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:39.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:40.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:39.973+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4512 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:40.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:39.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:40.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:39.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:40.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:39.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:40.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:39.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:40.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:40.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:40.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:40.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:40.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:40.598+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:40.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:40.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:40.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:40.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:40.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:40.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:41.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:40.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:41.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:40.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:41.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:40.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:41.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:40.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:41.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:41.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:41.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:41.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:41.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:41.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:41.912 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:41.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:41.912 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:41.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:41.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:41.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:42.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:41.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:42.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:41.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:42.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:41.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:42.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:41.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:42.906 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:42.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:42.906 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:42.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:42.906 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:42.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:42.906 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:42.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:42.906 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:42.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:42.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:42.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:43.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:42.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:43.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:42.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:43.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:42.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:43.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:42.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:43.864 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:43.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:43.865 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:43.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:43.865 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:43.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:43.865 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:43.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:43.865 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:43.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:43.865 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:43.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:44.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:43.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:44.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:43.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:44.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:43.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:44.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:43.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:44.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:44.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:44.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:44.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:44.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:44.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:44.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:44.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:44.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:44.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:44.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:44.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:45.111 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:44.974+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4613 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:44.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:44.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:44.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:44.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:45.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:45.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:45.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:45.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:45.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:45.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:45.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:45.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:45.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:45.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:45.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:45.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:46.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:45.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:46.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:45.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:46.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:45.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:46.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:45.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:46.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:46.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:46.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:46.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:46.823 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:46.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:46.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:46.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:46.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:46.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:46.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:46.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:47.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:46.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:47.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:46.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:47.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:46.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:47.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:46.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:47.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:47.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:47.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:47.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:47.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:47.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:47.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:47.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:47.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:47.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:47.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:47.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:48.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:47.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:48.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:47.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:48.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:47.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:48.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:47.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:48.914 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:48.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:48.914 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:48.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:48.914 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:48.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:48.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:48.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:48.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:48.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:48.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:48.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:49.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:48.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:49.262 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:48.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:49.262 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:48.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:49.262 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:48.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:49.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:49.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:49.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:49.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:49.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:49.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:49.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:49.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:49.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:49.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:49.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:49.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:50.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:49.974+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4696 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:50.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:49.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:50.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:49.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:50.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:49.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:50.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:49.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:50.900 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:50.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:50.901 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:50.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:50.901 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:50.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:50.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:50.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:50.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:50.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:50.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:50.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:51.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:50.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:51.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:50.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:51.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:50.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:51.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:50.889+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:51.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:51.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:51.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:51.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:51.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:51.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:51.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:51.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:51.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:52.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:51.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:52.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:51.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:52.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:51.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:52.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:51.868+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:52.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:52.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:52.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:52.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:52.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:52.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:52.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:52.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:52.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:52.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:52.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:52.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:53.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:52.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:53.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:52.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:53.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:52.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:53.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:52.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:53.860 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:53.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:53.861 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:53.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:53.861 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:53.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:53.861 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:53.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:53.861 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:53.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:53.861 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:53.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:54.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:53.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:54.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:53.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:54.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:53.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:54.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:53.849+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:54.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:54.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:54.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:54.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:54.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:54.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:54.869 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:54.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:54.869 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:54.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:54.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:54.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:55.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:54.975+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4778 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:20:55.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:54.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:55.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:54.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:55.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:54.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:55.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:54.857+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:55.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:55.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:55.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:55.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:55.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:55.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:55.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:55.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:55.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:55.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:55.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:55.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:56.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:55.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:56.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:55.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:56.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:55.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:56.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:55.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:56.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:56.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:56.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:56.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:56.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:56.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:56.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:56.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:56.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:56.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:56.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:56.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:57.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:56.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:57.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:56.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:57.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:56.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:57.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:56.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:57.889 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:57.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:57.890 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:57.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:57.890 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:57.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:57.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:57.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:57.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:57.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:57.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:57.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:58.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:57.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:58.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:57.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:58.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:57.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:58.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:57.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:58.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:58.583+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:58.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:58.583+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:58.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:58.583+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:58.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:58.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:58.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:58.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:58.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:58.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:59.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:58.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:59.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:58.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:59.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:58.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:20:59.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:58.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:59.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:59.619+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:59.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:59.619+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:59.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:20:59.619+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:20:59.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:59.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:59.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:59.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:20:59.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:20:59.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:00.238 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:20:59.977+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4860 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:00.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:59.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:00.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:59.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:00.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:59.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:00.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:20:59.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:00.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:00.607+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:00.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:00.607+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:00.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:00.607+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:00.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:00.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:00.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:00.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:00.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:00.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:01.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:00.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:01.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:00.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:01.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:00.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:01.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:00.908+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:01.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:01.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:01.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:01.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:01.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:01.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:01.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:01.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:01.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:01.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:01.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:01.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:02.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:01.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:02.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:01.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:02.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:01.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:02.239 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:01.954+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:02.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:02.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:02.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:02.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:02.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:02.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:02.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:02.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:02.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:02.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:02.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:02.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:03.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:02.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:03.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:02.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:03.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:02.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:03.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:02.915+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:03.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:03.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:03.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:03.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:03.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:03.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:03.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:03.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:03.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:03.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:03.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:03.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:04.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:03.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:04.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:03.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:04.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:03.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:04.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:03.923+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:04.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:04.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:04.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:04.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:04.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:04.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:04.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:04.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:04.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:04.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:04.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:04.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:05.237 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:04.978+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 4942 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:05.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:04.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:05.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:04.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:05.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:04.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:05.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:04.934+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:05.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:05.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:05.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:05.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:05.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:05.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:05.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:05.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:05.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:05.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:05.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:05.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:05.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:05.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:05.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:05.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:05.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:05.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:05.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:05.914+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:06.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:06.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:06.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:06.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:06.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:06.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:06.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:06.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:06.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:06.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:06.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:06.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:07.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:06.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:07.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:06.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:07.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:06.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:07.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:06.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:07.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:07.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:07.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:07.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:07.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:07.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:07.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:07.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:07.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:07.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:07.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:07.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:08.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:07.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:08.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:07.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:08.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:07.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:08.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:07.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:08.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:08.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:08.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:08.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:08.983 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:08.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:08.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:08.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:08.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:08.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:08.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:08.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:09.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:08.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:09.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:08.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:09.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:08.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:09.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:08.971+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:09.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:09.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:09.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:09.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:09.988 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:09.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:09.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:09.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:09.989 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:09.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:09.989 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:09.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:10.466 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:09.979+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5025 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:10.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:10.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:10.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:10.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.015+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:10.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:10.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:10.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:10.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:10.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:10.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:10.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:10.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:10.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:10.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:10.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:10.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:11.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:11.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:11.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:11.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:10.996+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:11.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:11.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:11.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:11.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:11.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:11.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:11.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:11.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:11.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:11.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:11.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:11.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:12.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:12.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:12.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:12.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:12.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:12.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:12.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:12.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:12.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:12.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:12.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:12.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:12.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:12.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:12.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:12.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:12.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:12.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:12.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:12.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:13.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:13.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:13.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:13.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:13.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:13.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:13.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:13.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:13.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:13.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:13.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:13.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:13.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:13.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:13.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:13.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:13.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:13.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:13.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:13.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:14.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:14.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:14.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:14.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:14.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:14.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:14.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:14.006+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:14.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:14.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:14.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:14.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:14.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:14.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:14.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:14.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:14.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:14.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:14.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:14.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:15.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:14.981+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5109 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:15.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:15.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:15.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:15.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:15.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:15.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:15.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:15.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:15.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:15.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:15.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:15.570+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:15.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:15.570+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:15.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:15.570+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:16.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:16.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:16.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:16.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:15.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:16.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:16.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:16.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:16.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:16.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:16.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:16.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:16.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:16.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:16.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:16.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:16.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:17.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:16.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:17.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:16.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:17.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:16.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:17.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:16.970+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:17.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:17.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:17.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:17.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:17.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:17.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:17.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:17.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:17.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:17.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:17.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:17.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:18.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:17.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:18.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:17.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:18.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:17.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:18.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:17.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:18.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:18.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:18.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:18.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:18.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:18.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:18.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:18.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:18.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:18.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:18.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:18.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:19.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:18.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:19.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:18.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:19.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:18.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:19.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:18.989+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:19.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:19.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:19.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:19.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:19.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:19.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:19.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:19.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:19.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:19.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:19.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:19.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:20.237 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:19.983+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5192 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:20.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:19.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:20.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:19.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:20.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:19.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:20.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:19.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:20.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:20.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:20.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:20.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:20.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:20.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:20.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:20.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:20.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:20.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:20.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:20.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:21.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:20.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:21.455 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:20.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:21.455 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:20.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:21.455 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:20.994+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:21.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:21.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:21.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:21.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:21.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:21.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:21.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:21.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:21.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:21.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:21.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:21.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:22.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:21.973+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:22.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:21.973+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:22.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:21.973+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:22.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:21.974+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:22.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:22.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:22.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:22.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:22.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:22.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:22.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:22.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:22.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:22.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:22.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:22.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:23.396 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:22.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:23.396 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:22.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:23.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:22.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:23.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:22.982+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:23.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:23.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:23.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:23.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:23.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:23.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:23.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:23.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:23.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:23.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:23.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:23.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:24.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:24.009+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:24.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:24.009+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:24.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:24.009+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:24.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:24.009+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:24.514+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:24.514+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:24.514+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:24.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:24.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:24.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:24.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:24.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:24.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:25.394 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:24.984+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5276 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:25.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:25.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:25.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:25.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:25.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:25.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:25.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:25.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:25.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:25.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:25.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:25.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:25.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:25.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:25.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:25.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:26.353 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:26.353 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:26.353 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:26.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:25.977+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:26.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:26.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:26.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:26.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:26.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:26.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:26.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:26.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:26.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:26.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:26.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:26.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:27.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:26.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:27.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:26.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:27.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:26.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:27.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:26.964+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:27.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:27.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:27.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:27.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:27.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:27.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:27.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:27.448+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:27.448+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:27.448+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:28.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:27.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:28.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:27.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:28.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:27.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:28.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:27.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:28.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:28.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:28.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:28.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:28.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:28.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:28.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:28.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:28.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:28.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:28.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:28.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:29.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:28.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:29.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:28.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:29.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:28.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:29.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:28.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:29.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:29.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:29.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:29.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:29.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:29.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:29.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:29.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:29.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:29.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:29.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:29.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:30.237 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:29.986+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5359 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:30.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:29.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:30.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:29.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:30.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:29.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:30.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:29.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:30.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:30.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:30.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:30.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:30.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:30.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:30.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:30.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:30.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:30.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:30.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:30.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:31.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:30.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:31.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:30.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:31.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:30.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:31.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:30.843+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:31.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:31.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:31.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:31.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:31.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:31.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:31.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:31.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:31.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:31.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:31.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:31.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:32.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:31.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:32.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:31.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:32.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:31.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:32.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:31.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:32.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:32.618+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:32.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:32.618+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:32.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:32.618+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:32.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:32.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:32.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:32.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:32.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:32.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:33.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:32.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:33.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:32.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:33.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:32.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:33.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:32.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:33.669 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:33.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:33.669 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:33.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:33.669 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:33.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:33.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:33.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:33.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:33.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:33.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:33.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:33.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:33.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:33.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:33.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:33.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:33.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:33.989 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:33.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:34.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:34.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:34.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:34.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:34.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:34.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:34.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:34.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:34.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:34.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:34.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:34.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:34.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:34.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:34.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:34.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:34.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:34.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:34.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:34.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:35.370 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:34.987+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5443 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:35.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:35.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:35.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:35.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:35.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:35.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:35.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:35.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:35.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:35.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:35.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:35.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:36.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:35.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:36.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:35.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:36.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:35.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:36.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:35.893+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:36.586 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:36.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:36.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:36.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:36.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:36.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:36.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:36.576+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:36.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:36.576+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:36.871 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:36.576+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:37.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:36.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:37.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:36.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:37.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:36.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:37.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:36.860+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:37.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:37.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:37.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:37.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:37.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:37.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:37.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:37.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:37.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:37.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:37.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:37.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:38.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:37.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:38.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:37.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:38.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:37.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:38.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:37.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:38.487 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:38.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:38.487 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:38.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:38.487 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:38.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:38.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:38.611+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:38.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:38.611+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:38.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:38.611+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:38.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:38.848+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:38.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:38.848+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:38.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:38.848+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:38.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:38.848+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:39.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:39.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:39.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:39.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:39.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:39.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:39.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:39.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:39.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:39.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:39.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:39.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:39.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:39.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:39.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:39.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:39.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:39.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:39.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:39.892+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:40.431 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:39.987+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5526 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:40.736 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:40.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:40.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:40.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:40.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:40.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:40.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:40.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:40.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:40.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:40.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:40.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:41.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:40.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:41.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:40.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:41.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:40.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:41.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:40.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:41.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:41.585+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:41.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:41.585+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:41.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:41.585+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:41.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:41.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:41.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:41.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:41.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:41.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:42.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:41.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:42.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:41.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:42.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:41.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:42.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:41.890+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:42.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:42.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:42.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:42.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:42.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:42.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:42.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:42.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:42.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:42.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:42.738 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:42.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:42.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:42.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:42.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:42.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:43.487 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:43.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:43.487 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:43.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:43.487 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:43.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:43.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:43.652+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:43.869 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:43.652+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:43.869 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:43.652+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:44.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:43.885+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:44.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:43.885+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:44.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:43.885+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:44.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:43.885+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:44.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:44.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:44.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:44.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:44.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:44.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:44.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:44.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:44.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:44.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:44.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:44.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:45.237 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:44.989+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5610 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:45.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:44.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:45.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:44.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:45.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:44.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:45.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:44.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:45.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:45.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:45.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:45.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:45.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:45.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:45.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:45.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:45.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:45.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:45.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:45.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:45.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:45.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:45.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:45.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:45.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:45.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:45.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:45.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:46.681 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:46.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:46.681 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:46.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:46.681 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:46.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:46.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:46.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:46.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:46.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:46.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:46.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:46.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:46.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:46.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:46.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:46.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:46.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:46.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:46.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:47.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:47.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:47.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:47.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:47.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:47.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:47.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:47.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:47.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:47.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:47.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:47.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:47.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:47.929+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:47.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:47.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:47.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:47.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:47.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:47.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:48.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:48.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:48.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:48.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:48.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:48.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:48.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:48.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:48.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:48.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:48.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:48.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:48.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:48.906+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:48.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:48.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:48.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:48.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:48.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:48.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:49.685 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:49.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:49.685 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:49.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:49.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:49.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:49.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:49.885+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:49.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:49.885+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:49.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:49.885+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:49.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:49.885+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:49.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:49.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:49.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:49.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:49.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:49.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:50.338 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:49.990+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5692 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:50.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:50.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:50.735 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:50.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:50.735 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:50.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:50.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:50.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:50.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:50.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:50.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:50.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:50.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:50.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:51.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:51.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:51.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:51.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:51.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:51.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:52.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:51.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:52.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:51.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:52.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:51.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:52.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:51.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:52.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:51.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:52.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:51.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:52.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:51.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:52.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:52.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:52.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:52.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:52.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:52.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:53.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:52.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:53.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:52.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:53.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:52.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:53.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:52.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:53.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:52.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:53.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:52.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:53.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:52.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:53.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:53.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:53.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:53.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:53.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:53.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:54.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:53.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:54.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:53.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:54.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:53.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:54.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:53.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:54.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:53.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:54.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:53.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:54.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:53.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:54.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:54.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:54.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:54.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:54.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:54.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:55.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:54.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:55.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:54.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:55.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:54.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:55.002 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:54.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:55.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:54.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:55.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:54.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:55.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:54.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:55.393 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:54.991+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5775 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:21:55.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:55.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:55.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:55.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:55.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:55.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:55.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:55.721+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:55.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:55.721+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:55.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:55.721+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:55.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:55.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:55.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:55.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:55.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:55.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:55.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:55.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:56.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:56.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:56.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:56.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:56.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:56.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:56.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:56.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:56.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:56.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:56.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:56.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:56.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:56.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:56.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:56.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:56.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:57.677 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:57.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:57.677 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:57.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:57.678 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:57.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:57.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:57.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:57.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:57.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:57.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:57.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:57.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:57.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:57.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:57.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:57.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:57.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:57.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:57.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:58.631 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:58.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:58.631 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:58.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:58.631 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:58.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:58.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:58.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:58.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:58.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:58.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:58.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:58.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:58.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:58.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:58.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:58.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:58.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:58.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:58.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:59.626 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:59.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:59.626 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:59.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:59.626 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:21:59.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:21:59.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:59.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:59.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:59.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:59.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:21:59.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:21:59.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:59.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:59.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:59.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:59.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:59.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:21:59.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:21:59.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:00.331 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:21:59.993+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5857 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:00.593 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:00.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:00.593 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:00.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:00.593 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:00.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:00.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:00.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:00.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:00.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:00.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:00.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:00.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:00.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:00.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:00.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:00.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:00.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:00.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:00.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:01.606 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:01.313+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:01.606 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:01.313+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:01.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:01.313+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:01.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:01.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:01.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:01.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:01.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:01.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:01.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:01.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:01.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:01.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:01.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:01.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:01.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:01.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:02.590 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:02.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:02.590 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:02.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:02.590 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:02.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:02.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:02.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:02.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:02.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:02.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:02.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:02.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:02.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:03.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:02.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:03.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:02.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:03.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:02.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:03.584 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:03.272+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:03.584 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:03.272+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:03.584 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:03.272+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:03.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:03.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:03.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:03.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:03.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:03.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:03.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:03.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:04.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:03.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:04.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:03.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:04.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:03.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:04.616 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:04.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:04.616 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:04.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:04.616 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:04.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:04.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:04.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:04.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:04.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:04.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:04.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:04.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:04.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:04.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:04.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:04.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:04.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:04.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:04.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:05.336 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:04.995+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 5940 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:05.589 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:05.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:05.590 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:05.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:05.590 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:05.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:05.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:05.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:05.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:05.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:05.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:05.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:05.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:05.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:06.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:05.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:06.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:05.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:06.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:05.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:06.620 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:06.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:06.620 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:06.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:06.620 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:06.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:06.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:06.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:06.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:06.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:06.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:06.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:06.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:06.611+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:07.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:06.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:07.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:06.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:07.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:06.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:07.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:07.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:07.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:07.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:07.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:07.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:07.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:07.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:07.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:07.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:07.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:07.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:07.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:07.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:07.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:07.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:07.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:07.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:07.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:07.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:08.665 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:08.309+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:08.665 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:08.309+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:08.665 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:08.309+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:08.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:08.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:08.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:08.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:08.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:08.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:08.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:08.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:08.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:08.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:08.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:08.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:08.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:08.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:09.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:09.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:09.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:09.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:09.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:09.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:09.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:09.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:09.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:09.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:09.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:09.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:09.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:09.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:09.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:09.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:09.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:09.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:09.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:09.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:10.312 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:09.996+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6022 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:10.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:10.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:10.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:10.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:10.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:10.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:10.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:10.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:10.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:10.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:10.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:10.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:10.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:10.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:10.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:10.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:10.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:10.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:10.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:10.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:11.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:11.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:11.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:11.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:11.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:11.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:11.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:11.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:11.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:11.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:11.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:11.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:11.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:11.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:11.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:11.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:11.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:11.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:11.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:11.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:12.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:12.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:12.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:12.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:12.638 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:12.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:12.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:12.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:12.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:12.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:12.889 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:12.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:12.889 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:12.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:13.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:12.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:13.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:12.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:13.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:12.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:13.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:13.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:13.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:13.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:13.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:13.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:13.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:13.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:13.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:13.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:13.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:13.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:13.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:13.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:14.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:13.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:14.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:13.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:14.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:13.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:14.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:14.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:14.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:14.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:14.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:14.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:14.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:14.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:14.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:14.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:14.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:14.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:14.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:14.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:15.236 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:14.998+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6105 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:15.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:14.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:15.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:14.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:15.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:14.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:15.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:15.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:15.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:15.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:15.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:15.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:15.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:15.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:15.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:15.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:15.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:15.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:15.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:15.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:16.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:15.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:16.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:15.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:16.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:15.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:16.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:16.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:16.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:16.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:16.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:16.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:16.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:16.639+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:16.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:16.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:16.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:16.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:16.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:16.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:17.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:16.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:17.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:16.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:17.237 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:16.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:17.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:17.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:17.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:17.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:17.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:17.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:17.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:17.659+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:17.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:17.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:17.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:17.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:17.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:17.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:17.872 INFO:teuthology.orchestra.run.smithi039.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:22:17.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-02-19T19:22:17.861+0000 7f6e484ad700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:22:18.169 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:17.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:18.169 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:17.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:18.169 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:17.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:22:18.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:18.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:18.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:18.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:18.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:18.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:18.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:18.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:18.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:18.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:18.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:18.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:18.737 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:18.697+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:19.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:18.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:19.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:18.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:19.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:18.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:19.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:19.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:19.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:19.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:19.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:19.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:19.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:19.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:19.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:19.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:19.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:19.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:19.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:19.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:20.137 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:19.999+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6187 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:20.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:19.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:20.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:19.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:20.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:19.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:20.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:20.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:20.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:20.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:20.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:20.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:20.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:20.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:20.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:20.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:20.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:20.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:20.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:20.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:21.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:20.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:21.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:20.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:21.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:20.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:21.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:21.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:21.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:21.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:21.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:21.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:21.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:21.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:21.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:21.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:21.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:21.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:21.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:21.706+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:22.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:21.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:22.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:21.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:22.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:21.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:22.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:22.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:22.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:22.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:22.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:22.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:22.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:22.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:22.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:22.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:22.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:22.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:22.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:22.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:23.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:22.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:23.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:22.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:23.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:22.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:23.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:23.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:23.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:23.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:23.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:23.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:23.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:23.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:23.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:23.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:23.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:23.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:23.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:23.778+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:23.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:23.778+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:23.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:23.778+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:23.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:23.778+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:24.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:24.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:24.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:24.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:24.987 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:24.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:24.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:24.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:24.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:24.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:24.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:24.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:24.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:24.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:24.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:24.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:24.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:24.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:24.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:24.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:25.486 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:25.000+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6271 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:25.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:25.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:25.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:25.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:25.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:25.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:25.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:25.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:25.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:25.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:25.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:25.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:25.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:25.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:26.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:25.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:26.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:25.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:26.236 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:25.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:26.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:26.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:26.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:26.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:26.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:26.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:26.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:26.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:26.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:26.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:26.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:26.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:26.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:26.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:26.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:26.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:26.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:26.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:26.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:26.823+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:27.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:27.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:27.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:27.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:27.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:27.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:27.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:27.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:02.495600+0000 front 2024-02-19T19:12:02.495580+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:27.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:27.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:02.495532+0000 front 2024-02-19T19:12:02.495633+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:27.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:27.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:02.495615+0000 front 2024-02-19T19:12:02.495666+0000 (oldest deadline 2024-02-19T19:12:24.195522+0000) 2024-02-19T19:22:27.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:27.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:27.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:27.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:27.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:27.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:27.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:27.837+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:28.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:28.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:11:58.235934+0000 front 2024-02-19T19:11:58.236105+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:28.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:28.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:11:58.235952+0000 front 2024-02-19T19:11:58.236141+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:28.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:28.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:11:58.236200+0000 front 2024-02-19T19:11:58.235990+0000 (oldest deadline 2024-02-19T19:12:24.135928+0000) 2024-02-19T19:22:28.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:28.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:28.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:28.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:28.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:28.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:28.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:28.820+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:30.140 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:30.001+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6353 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:30.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:29.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:30.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:29.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:30.141 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:29.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:30.141 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:29.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:31.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:30.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:31.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:30.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:31.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:30.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:31.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:30.747+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:31.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:31.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:31.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:31.764+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:32.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:32.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:32.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:32.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:32.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:32.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:32.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:32.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:33.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:33.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:33.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:33.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:33.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:33.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:33.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:33.757+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:35.013 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:34.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:35.014 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:34.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:35.014 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:34.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:35.014 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:34.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:35.486 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:35.003+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6437 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:35.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:35.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:35.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:35.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:35.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:35.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:35.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:35.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:37.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:36.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:37.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:36.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:37.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:36.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:37.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:36.755+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:37.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:37.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:37.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:37.774+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:38.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:38.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:38.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:38.759+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:40.139 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:40.004+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6519 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:40.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:39.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:40.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:39.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:40.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:39.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:40.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:39.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:41.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:40.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 since back 2024-02-19T19:12:03.065973+0000 front 2024-02-19T19:12:03.065869+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:41.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:40.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-02-19T19:12:03.065947+0000 front 2024-02-19T19:12:03.065983+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:41.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:40.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 since back 2024-02-19T19:12:03.065935+0000 front 2024-02-19T19:12:03.065960+0000 (oldest deadline 2024-02-19T19:12:27.765855+0000) 2024-02-19T19:22:41.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:40.811+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:42.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:41.824+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:42.832+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:44.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:43.832+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:45.139 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:45.006+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6602 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:45.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:44.864+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:45.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:47.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:46.879+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:48.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:47.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:49.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:48.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:50.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:49.904+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:50.236 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:50.007+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6684 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:50.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:50.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:50.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:50.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:50.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:50.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:50.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:50.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:51.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:51.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:51.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:51.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:51.737 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:51.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:52.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:51.920+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:52.986 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:52.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:52.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:52.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:52.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:52.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:52.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:52.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:52.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:52.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:52.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:52.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:52.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:52.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:53.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:53.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:53.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:53.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:53.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:53.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:53.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:53.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:53.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:53.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:53.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:53.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:54.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:53.845+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:54.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:54.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:54.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:54.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:54.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:54.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:55.142 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:22:55.010+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6767 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:22:55.143 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:54.804+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:55.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:54.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:55.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:54.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:55.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:54.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:55.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:55.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:55.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:55.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:55.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:55.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:55.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:55.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:55.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:55.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:55.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:55.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:55.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:55.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:56.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:56.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:56.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:56.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:56.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:56.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:56.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:56.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:56.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:56.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:56.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:56.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:56.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:56.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:57.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:57.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:57.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:57.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:57.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:57.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:57.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:57.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:57.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:57.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:57.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:57.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:57.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:57.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:58.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:58.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:58.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:58.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:58.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:58.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:58.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:58.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:58.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:58.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:58.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:58.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:58.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:58.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:59.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:22:59.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:59.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:59.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:59.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:59.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:59.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:22:59.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:22:59.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:59.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:59.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:59.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:22:59.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:22:59.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:00.486 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:00.011+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6849 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:00.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:00.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:00.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:00.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:00.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:00.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:00.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:00.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:00.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:00.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:00.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:00.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:00.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:00.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:01.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:01.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:01.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:01.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:01.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:01.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:01.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:01.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:01.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:01.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:01.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:01.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:01.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:01.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:02.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:02.864+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:02.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:02.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:02.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:02.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:02.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:02.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:02.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:02.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:02.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:02.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:02.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:02.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:03.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:03.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:03.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:03.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:03.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:03.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:03.907 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:03.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:03.907 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:03.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:03.908 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:03.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:03.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:03.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:03.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:04.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:03.898+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:04.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:04.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:04.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:04.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:04.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:04.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:04.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:04.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:04.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:04.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:04.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:04.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:05.235 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:05.013+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 6932 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:05.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:04.945+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:05.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:04.945+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:05.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:04.945+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:05.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:04.945+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:05.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:05.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:05.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:05.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:05.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:05.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:05.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:05.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:05.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:05.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:05.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:05.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:05.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:05.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:05.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:05.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:06.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:06.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:06.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:06.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:06.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:06.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:06.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:06.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:06.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:06.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:06.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:06.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:07.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:06.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:07.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:06.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:07.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:06.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:07.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:06.998+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:07.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:07.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:07.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:07.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:07.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:07.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:07.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:07.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:07.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:08.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:07.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:08.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:07.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:08.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:07.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:08.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:07.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:08.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:08.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:08.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:08.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:08.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:08.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:08.969+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:08.969+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:08.969+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:08.969+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:09.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:09.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:09.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:09.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:09.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:09.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:09.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:09.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:09.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:09.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:09.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:09.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:10.235 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:10.014+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7014 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:10.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:09.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:10.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:09.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:10.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:09.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:10.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:09.950+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:10.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:10.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:10.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:10.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:10.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:10.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:10.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:10.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:10.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:10.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:10.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:10.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:11.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:11.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:11.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:11.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:11.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:11.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:11.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:11.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:11.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:11.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:11.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:11.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:11.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:11.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:11.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:11.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:11.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:11.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:11.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:11.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:12.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:12.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:12.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:12.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:12.039+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:12.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:12.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:12.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:12.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:12.956 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:12.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:12.956 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:12.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:12.956 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:12.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:12.956 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:12.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:13.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:13.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:13.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:13.085+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:13.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:13.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:13.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:13.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:13.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:13.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:13.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:13.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:13.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:13.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:13.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:13.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:14.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:14.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:14.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:14.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:14.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:14.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:14.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:14.052+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:14.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:14.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:14.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:14.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:14.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:14.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:14.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:14.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:14.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:14.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:14.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:14.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:15.485 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:15.015+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7098 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:15.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:15.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:15.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:15.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.032+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:15.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:15.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:15.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:15.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:15.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:15.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:15.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:15.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:15.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:15.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:15.987 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:15.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:15.996+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:16.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:16.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:16.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:16.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:16.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:16.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:16.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:16.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:16.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:16.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:16.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:16.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:16.976+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:16.976+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:16.976+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:16.976+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:17.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:17.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:17.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:17.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:17.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:17.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:17.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:17.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:17.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:17.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:17.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:17.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.019+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:18.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:18.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:18.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:18.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:18.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:18.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:18.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:18.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:18.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:18.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:18.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:18.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:19.278 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:19.278 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:19.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:19.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:18.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:19.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:19.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:19.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:19.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:19.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:19.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:19.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:19.549+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:19.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:19.549+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:19.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:19.549+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:20.485 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:20.016+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7181 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:20.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:19.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:20.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:19.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:20.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:19.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:20.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:19.982+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:20.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:20.596+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:20.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:20.596+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:20.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:20.596+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:20.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:20.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:20.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:20.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:20.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:20.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:21.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:21.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:21.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:21.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:21.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:21.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:21.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:21.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:21.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:21.614+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:21.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:21.614+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:21.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:21.614+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:21.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:21.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:21.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:21.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:21.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:21.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:22.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:22.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:22.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:22.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:22.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:22.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:22.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:22.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:22.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:22.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:22.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:22.540+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:22.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:22.540+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:22.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:22.540+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:22.986+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:23.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:23.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:23.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:23.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:23.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:23.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:23.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:23.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:23.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:23.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:23.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:23.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:24.334 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:24.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:24.334 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:24.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:24.334 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:24.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:24.335 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:24.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:24.626 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:24.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:24.626 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:24.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:24.627 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:24.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:24.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:24.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:24.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:24.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:24.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:24.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:25.485 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:25.018+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7264 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:25.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:25.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:25.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:25.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:25.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:25.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:25.486 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:25.050+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:25.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:25.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:25.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:25.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:25.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:25.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:25.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:25.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:25.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:25.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:25.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:25.659+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:26.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:26.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:26.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:26.098+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:26.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:26.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:26.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:26.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:26.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:26.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:26.821 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:26.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:26.821 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:26.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:26.821 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:26.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:27.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:27.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:27.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:27.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:27.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:27.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:27.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:27.075+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:27.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:27.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:27.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:27.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:27.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:27.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:27.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:27.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:27.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:27.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:27.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:27.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:28.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:28.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:28.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:28.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:28.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:28.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:28.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:28.041+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:28.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:28.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:28.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:28.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:28.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:28.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:29.012 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:28.750+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:29.012 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:28.750+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:29.012 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:28.750+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:29.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:29.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:29.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:29.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:29.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:29.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:29.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:29.003+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:29.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:29.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:29.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:29.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:29.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:29.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:30.153 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:30.019+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7347 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:30.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:29.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:30.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:29.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:30.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:29.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:30.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:30.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:30.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:30.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.002+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:30.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:30.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:30.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:30.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:30.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:30.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:30.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:31.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:30.826+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:31.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:30.826+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:31.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:30.826+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:31.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:31.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:31.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:31.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:31.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:31.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:32.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:31.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:32.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:31.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:32.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:31.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:32.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:31.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:31.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:31.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:32.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:31.985+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:32.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:32.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:32.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:32.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:32.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:32.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:33.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:33.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:33.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:33.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:32.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:33.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:32.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:33.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:32.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:33.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:32.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:33.831 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:33.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:33.832 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:33.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:33.832 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:33.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:34.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:33.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:34.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:33.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:34.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:33.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:34.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:34.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:34.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:34.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:34.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:34.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:34.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:34.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:34.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:34.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:34.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:34.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:34.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:34.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:35.041 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:35.021+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7430 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:35.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:34.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:35.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:34.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:35.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:34.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:35.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:35.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:35.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:35.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:35.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:35.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:35.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:35.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:35.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:35.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:35.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:35.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:35.985 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:35.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:35.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:35.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:35.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:35.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:35.986 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:35.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:36.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:36.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:36.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:36.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:36.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:36.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:36.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:36.080+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:36.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:36.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:36.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:37.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:36.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:37.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:36.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:37.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:36.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:37.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:37.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:37.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:37.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:37.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:37.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:37.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:37.078+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:37.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:37.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:37.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:37.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:37.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:37.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:38.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:37.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:38.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:37.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:38.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:37.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:38.038+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:38.038+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:38.038+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:38.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:38.038+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:38.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:38.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:38.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:38.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:38.871 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:38.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:39.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:38.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:39.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:38.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:39.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:38.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:39.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.027+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.027+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.027+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:39.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.028+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:39.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:39.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:39.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:39.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:39.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:39.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:40.157 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:40.024+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7512 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:40.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:39.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:40.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:39.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:40.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:39.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:40.158 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:40.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:40.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:40.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:39.992+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:40.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:40.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:40.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:40.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:40.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:40.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:41.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:40.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:41.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:40.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:41.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:40.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:41.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:40.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:41.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:40.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:41.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:40.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:41.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:40.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:41.923 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:41.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:41.923 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:41.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:41.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:41.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:42.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:41.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:42.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:41.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:42.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:41.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:42.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:41.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:42.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:41.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:42.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:41.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:42.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:41.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:42.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:42.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:42.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:42.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:42.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:42.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:43.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:43.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:43.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:43.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:43.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:42.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:43.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:42.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:43.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:42.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:43.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:43.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:43.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:43.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:43.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:43.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:44.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:43.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:44.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:43.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:44.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:43.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:44.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:44.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:44.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:44.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:44.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:44.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:44.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:44.047+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:44.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:44.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:44.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:44.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:44.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:44.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:45.235 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:45.025+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7595 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:45.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:44.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:45.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:44.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:45.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:44.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:45.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:45.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:45.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:45.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:45.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:45.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:45.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:45.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:45.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:45.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:46.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:45.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:46.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:45.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:46.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:45.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:45.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:46.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:46.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:46.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:46.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:46.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:46.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:47.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:46.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:47.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:46.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:47.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:46.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:47.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:46.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:47.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:46.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:47.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:46.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:47.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:46.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:47.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:47.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:47.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:47.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:47.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:47.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:48.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:47.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:48.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:47.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:48.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:47.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:48.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:47.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:48.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:47.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:48.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:47.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:48.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:47.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:48.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:48.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:48.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:48.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:48.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:48.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:49.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:48.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:49.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:48.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:49.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:48.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:49.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:48.880+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:49.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:48.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:49.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:48.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:49.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:48.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:49.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:49.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:49.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:49.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:49.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:49.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:50.158 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:50.027+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7677 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:50.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:49.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:50.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:49.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:50.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:49.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:50.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:49.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:50.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:49.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:50.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:49.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:50.160 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:49.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:50.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:50.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:50.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:50.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:50.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:50.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:51.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:50.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:51.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:50.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:51.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:50.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:51.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:50.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:51.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:50.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:51.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:50.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:51.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:50.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:51.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:51.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:51.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:51.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:51.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:51.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:52.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:51.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:52.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:51.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:52.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:51.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:52.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:51.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:52.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:51.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:52.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:51.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:52.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:51.910+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:52.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:52.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:52.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:52.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:52.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:52.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:53.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:52.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:53.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:52.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:53.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:52.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:53.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:52.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:53.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:52.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:53.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:52.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:53.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:52.929+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:53.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:53.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:53.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:53.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:53.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:53.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:54.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:53.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:54.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:53.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:54.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:53.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:54.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:53.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:54.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:53.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:54.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:53.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:54.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:53.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:54.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:54.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:54.673 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:54.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:54.673 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:54.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:55.235 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:23:55.029+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7760 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:23:55.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:54.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:55.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:54.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:55.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:54.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:55.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:54.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:55.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:54.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:55.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:54.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:55.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:54.942+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:55.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:55.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:55.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:55.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:55.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:55.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:56.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:56.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:56.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:56.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:56.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:56.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:56.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:55.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:56.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:55.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:56.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:55.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:56.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:55.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:56.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:56.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:56.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:56.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:56.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:56.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:57.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:56.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:57.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:56.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:57.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:56.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:57.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:56.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:57.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:56.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:57.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:56.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:57.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:56.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:57.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:57.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:57.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:57.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:57.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:57.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:58.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:58.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:58.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:58.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:58.236 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:58.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:58.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:57.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:58.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:57.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:58.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:57.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:58.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:57.944+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:58.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:58.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:58.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:58.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:58.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:58.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:59.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:58.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:59.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:58.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:59.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:23:58.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:23:59.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:58.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:59.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:58.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:59.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:58.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:59.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:58.956+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:59.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:59.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:23:59.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:23:59.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:59.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:59.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:59.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:59.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:59.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:23:59.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:23:59.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:00.420 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:00.029+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7842 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:00.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:00.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:00.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:00.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:00.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:00.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:00.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:00.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:00.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:00.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:00.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:00.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:01.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:01.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:01.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:01.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:01.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:01.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:01.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:00.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:01.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:00.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:01.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:00.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:01.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:00.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:01.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:01.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:01.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:01.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:01.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:01.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:02.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:02.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:02.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:02.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:02.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:02.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:02.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:01.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:02.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:01.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:02.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:01.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:02.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:01.936+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:02.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:02.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:02.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:02.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:02.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:02.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:03.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:03.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:03.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:03.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:03.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:03.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:03.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:02.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:03.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:02.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:03.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:02.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:03.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:02.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:03.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:03.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:03.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:03.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:03.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:03.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:04.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:04.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:04.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:04.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:04.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:04.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:03.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:03.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:03.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:04.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:03.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:04.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:04.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:04.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:04.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:04.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:04.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:05.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:04.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:05.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:04.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:05.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:04.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:05.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:04.886+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:05.336 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:05.031+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 7924 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:05.336 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:05.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:05.336 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:05.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:05.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:05.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:05.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:05.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:05.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:05.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:05.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:05.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:06.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:06.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:06.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:06.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:06.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:06.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:05.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:05.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:05.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:06.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:05.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:06.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:06.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:06.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:06.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:06.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:06.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:07.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:06.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:07.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:06.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:07.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:06.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:07.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:06.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:07.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:07.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:07.392 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:07.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:07.392 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:07.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:07.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:07.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:07.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:07.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:07.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:07.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:08.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:08.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:08.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:08.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:08.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:08.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:08.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:07.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:08.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:07.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:08.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:07.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:08.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:07.894+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:08.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:08.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:08.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:08.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:08.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:08.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:09.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:09.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:09.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:09.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:09.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:09.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:08.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:08.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:08.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:09.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:08.862+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:09.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:09.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:09.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:09.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:09.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:09.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:10.113 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:10.032+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8007 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:10.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:09.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:09.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:09.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:09.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:10.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:10.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:10.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:10.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:10.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:10.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:10.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:10.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:10.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:10.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:10.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:10.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:11.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:11.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:11.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:11.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:11.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:11.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:11.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:10.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:11.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:10.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:11.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:10.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:11.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:10.923+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:11.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:11.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:11.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:11.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:11.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:11.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:12.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:11.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:12.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:11.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:12.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:11.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:12.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:11.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:12.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:12.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:12.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:12.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:12.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:12.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:12.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:12.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:12.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:12.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:12.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:12.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:13.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:13.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:13.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:13.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:13.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:13.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:12.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:12.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:12.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:13.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:12.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:13.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:13.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:13.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:13.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:13.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:13.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:14.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:14.106+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:14.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:14.106+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:14.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:14.106+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:14.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:13.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:14.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:13.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:14.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:13.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:14.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:13.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:14.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:14.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:14.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:14.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:14.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:14.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:15.169 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:15.034+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8089 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:15.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:15.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:15.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:15.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:15.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:15.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:15.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:14.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:15.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:14.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:15.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:14.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:15.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:14.909+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:15.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:15.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:15.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:15.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:15.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:15.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:16.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:16.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:16.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:16.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:16.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:16.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:16.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:15.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:16.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:15.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:16.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:15.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:16.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:15.959+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:16.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:16.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:16.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:16.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:16.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:16.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:17.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:17.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:17.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:17.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:17.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:17.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:16.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:16.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:16.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:17.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:16.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:17.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:17.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:17.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:17.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:17.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:17.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:18.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:18.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:18.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:18.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:18.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:18.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:18.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:17.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:18.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:17.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:18.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:17.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:18.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:17.962+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:18.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:18.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:18.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:18.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:18.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:18.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:19.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:18.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:19.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:18.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:19.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:18.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:19.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:18.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:19.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:18.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:19.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:18.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:19.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:18.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:19.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:19.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:19.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:19.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:19.735 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:19.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:20.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:20.035+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8172 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:20.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:19.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:20.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:19.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:20.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:19.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:20.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:19.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:20.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:19.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:20.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:19.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:20.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:19.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:20.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:20.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:20.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:20.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:20.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:20.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:21.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:20.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:21.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:20.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:21.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:20.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:21.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:20.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:21.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:20.921+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:21.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:20.921+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:21.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:20.921+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:21.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:21.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:21.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:21.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:21.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:21.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:22.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:21.885+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:22.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:21.885+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:22.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:21.885+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:22.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:21.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:22.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:21.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:22.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:21.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:22.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:21.971+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:22.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:22.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:22.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:22.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:22.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:22.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:23.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:22.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:23.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:22.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:23.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:22.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:23.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:22.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:23.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:22.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:23.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:22.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:23.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:22.944+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:23.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:23.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:23.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:23.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:23.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:23.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:24.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:23.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:24.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:23.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:24.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:23.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:24.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:23.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:24.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:23.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:24.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:23.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:24.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:23.962+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:24.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:24.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:24.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:24.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:24.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:24.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:25.169 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:25.038+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8254 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:25.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:24.900+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:25.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:24.900+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:25.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:24.900+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:25.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:24.973+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:25.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:24.973+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:25.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:24.973+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:25.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:24.973+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:25.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:25.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:25.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:25.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:25.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:25.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:26.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:25.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:26.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:25.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:26.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:25.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:26.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:25.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:26.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:25.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:26.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:25.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:26.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:25.952+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:26.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:26.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:26.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:26.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:26.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:26.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:27.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:26.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:27.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:26.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:27.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:26.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:27.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:26.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:27.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:26.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:27.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:26.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:27.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:26.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:27.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:27.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:27.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:27.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:27.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:27.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:28.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:27.839+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:28.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:27.839+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:28.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:27.839+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:28.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:28.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:28.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:28.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:28.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:28.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:28.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:28.023+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:28.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:28.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:28.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:28.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:28.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:28.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:29.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:28.832+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:29.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:28.832+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:29.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:28.832+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:29.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:29.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:29.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:29.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:29.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:29.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:29.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:29.052+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:29.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:29.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:29.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:30.160 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:30.039+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8331 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:30.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:29.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:30.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:29.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:30.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:29.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:30.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:30.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:30.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:30.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:30.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:30.309+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:30.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:30.309+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:30.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:30.309+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:31.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:30.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:31.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:30.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:31.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:30.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:31.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:31.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:30.957+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:31.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:31.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:31.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:31.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:31.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:31.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:32.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:31.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:32.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:31.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:32.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:31.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:32.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:31.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:32.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:31.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:32.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:31.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:32.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:31.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:32.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:32.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:32.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:32.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:32.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:32.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:33.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:32.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:33.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:32.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:33.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:32.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:33.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:32.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:33.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:32.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:33.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:32.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:33.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:32.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:33.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:33.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:33.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:33.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:33.592 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:33.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:33.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:33.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:33.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:33.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:33.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:33.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:34.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:33.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:34.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:33.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:34.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:33.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:34.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:33.981+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:34.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:34.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:34.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:34.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:34.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:34.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:35.127 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:35.041+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8407 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:35.127 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:34.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:35.127 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:34.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:35.127 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:34.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:35.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:34.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:35.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:34.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:35.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:34.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:35.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:34.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:35.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:35.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:35.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:35.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:35.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:35.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:36.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:35.870+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:36.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:35.870+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:36.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:35.870+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:36.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:36.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:36.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:36.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.001+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:36.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:36.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:36.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:36.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:36.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:36.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:37.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:36.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:37.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:36.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:37.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:36.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:37.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.966+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:37.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:37.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:37.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:36.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:37.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:37.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:37.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:37.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:37.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:37.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:38.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:37.895+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:38.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:37.895+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:38.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:37.895+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:38.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:37.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:38.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:37.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:38.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:37.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:38.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:37.989+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:38.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:38.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:38.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:38.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:38.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:38.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:39.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:38.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:39.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:38.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:39.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:38.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:39.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:38.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:39.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:38.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:39.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:38.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:39.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:38.984+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:39.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:39.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:39.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:39.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:39.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:39.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:40.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:40.043+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8484 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:40.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:39.924+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:40.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:39.924+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:40.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:39.924+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:40.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:40.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:40.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:40.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:40.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:40.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:40.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:40.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:40.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:40.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:40.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:40.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:40.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:40.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:41.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:40.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:41.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:40.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:41.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:40.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:41.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:41.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:41.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:41.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:41.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:41.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:41.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:41.071+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:41.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:41.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:41.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:41.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:41.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:41.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:42.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:41.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:42.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:41.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:42.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:41.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:42.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:42.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:42.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:42.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:42.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:42.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:42.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:42.087+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:42.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:42.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:42.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:42.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:42.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:42.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:43.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:42.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:43.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:42.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:43.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:42.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:43.107+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:43.107+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:43.107+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:43.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:43.108+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:43.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:43.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:43.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:43.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:43.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:43.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:44.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:43.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:44.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:43.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:44.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:43.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:44.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:44.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:44.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:44.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:44.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:44.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:44.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:44.089+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:44.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:44.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:44.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:44.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:44.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:44.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:45.234 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:45.044+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8557 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:45.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:44.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:45.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:44.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:45.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:44.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:45.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:45.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:45.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:45.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:45.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:45.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:45.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:45.055+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:45.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:45.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:45.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:45.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:45.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:45.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:46.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:45.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:46.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:45.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:46.235 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:45.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:46.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:46.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:46.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:46.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:46.095+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:46.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:46.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:46.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:46.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:46.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:46.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:47.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:46.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:47.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:46.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:47.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:46.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:47.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:47.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:47.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:47.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:47.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:47.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:47.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:47.130+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:47.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:47.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:47.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:47.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:47.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:47.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:48.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:48.129+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:48.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:48.129+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:48.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:48.129+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:48.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:48.129+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:48.363 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:47.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:48.363 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:47.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:48.364 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:47.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:48.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:48.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:48.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:48.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:48.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:48.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:49.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:49.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:49.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:49.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:49.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:49.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:49.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:49.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:49.412 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:49.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:49.412 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:49.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:49.412 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:49.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:49.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:49.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:49.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:49.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:49.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:49.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:50.453 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:50.045+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8631 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:50.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:50.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:50.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:50.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:50.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:50.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:50.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:50.142+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:50.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:50.142+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:50.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:50.142+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:50.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:50.142+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:50.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:50.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:50.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:50.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:50.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:50.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:51.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:51.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:51.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:51.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:51.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:51.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:51.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:51.152+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:51.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:51.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:51.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:51.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:51.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:51.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:51.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:51.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:51.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:51.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:51.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:51.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:52.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:52.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:52.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:52.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:52.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:52.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:52.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:52.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:52.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:52.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:52.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:52.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:52.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:52.115+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:52.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:52.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:52.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:52.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:52.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:52.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:53.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:53.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:53.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:53.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:53.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:53.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:53.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:53.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:53.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:53.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:53.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:53.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:53.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:53.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:53.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:53.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:53.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:53.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:53.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:53.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:54.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:54.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:54.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:54.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:54.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:54.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:54.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:54.105+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:54.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:54.146+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:54.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:54.146+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:54.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:54.146+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:54.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:54.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:54.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:54.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:54.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:54.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:55.346 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:24:55.046+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8710 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:24:55.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:55.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:55.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:55.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:55.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:55.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:55.347 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:55.133+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:55.347 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:55.133+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:55.347 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:55.133+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:55.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:55.133+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:55.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:55.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:55.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:55.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:55.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:55.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:56.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:56.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:56.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:56.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:56.125+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:56.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:56.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:56.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:56.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:56.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:56.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:56.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:56.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:56.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:56.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:56.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:56.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:57.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:57.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:57.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:57.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:57.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:57.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:57.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:57.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:57.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:57.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:57.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:57.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:57.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:57.116+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:57.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:57.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:57.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:57.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:57.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:57.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:58.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:58.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:58.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:58.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:58.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:58.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:58.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:58.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:58.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:58.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:58.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:58.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:58.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:58.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:58.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:59.458 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:59.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:59.458 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:59.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:59.458 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:24:59.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:24:59.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:59.185+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:59.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:59.185+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:59.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:59.185+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:24:59.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:24:59.185+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:59.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:59.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:59.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:59.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:24:59.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:24:59.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:00.402 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:00.048+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8790 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:00.051+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:00.051+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:00.051+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:00.145+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:00.145+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:00.145+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:00.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:00.145+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:00.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:00.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:00.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:00.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:00.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:00.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:01.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:01.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:01.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:01.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:01.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:01.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:01.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:01.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:01.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:01.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:01.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:01.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:01.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:01.148+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:01.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:01.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:01.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:01.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:01.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:01.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:02.447 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:02.119+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:02.447 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:02.119+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:02.447 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:02.119+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:02.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:02.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:02.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:02.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:02.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:02.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:02.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:02.175+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:02.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:02.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:02.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:02.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:02.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:02.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:03.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:03.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:03.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:03.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:03.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:03.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:03.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:03.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:03.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:03.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:03.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:03.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:03.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:03.156+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:03.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:03.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:03.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:03.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:03.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:03.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:04.415 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:04.152+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:04.415 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:04.152+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:04.415 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:04.152+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:04.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:04.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:04.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:04.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:04.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:04.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:04.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:04.177+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:04.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:04.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:04.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:04.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:04.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:04.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:05.404 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:05.050+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8872 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:05.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:05.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:05.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:05.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:05.405 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:05.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:05.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:05.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:05.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:05.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:05.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:05.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:05.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:05.190+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:05.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:05.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:05.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:05.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:05.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:05.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:06.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:06.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:06.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:06.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:06.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:06.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:06.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:06.222+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:06.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:06.222+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:06.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:06.222+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:06.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:06.222+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:06.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:06.164+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:06.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:06.164+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:06.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:06.164+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:07.442 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:07.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:07.442 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:07.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:07.442 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:07.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:07.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:07.202+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:07.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:07.202+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:07.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:07.202+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:07.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:07.202+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:07.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:07.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:07.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:07.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:07.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:07.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:08.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:08.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:08.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:08.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:08.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:08.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:08.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:08.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:08.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:08.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:08.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:08.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:08.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:08.187+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:08.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:08.187+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:08.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:08.187+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:08.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:08.187+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:09.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:09.209+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:09.470 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:09.209+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:09.470 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:09.209+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:09.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:09.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:09.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:09.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:09.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:09.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:09.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:09.175+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:09.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:09.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:09.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:09.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:09.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:09.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:10.429 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:10.051+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 8955 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:10.430 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:10.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:10.430 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:10.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:10.430 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:10.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:10.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:10.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:10.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:10.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:10.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:10.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:10.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:10.159+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:10.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:10.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:10.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:10.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:10.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:10.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:11.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:11.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:11.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:11.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:11.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:11.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:11.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:11.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:11.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:11.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:11.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:11.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:11.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:11.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:11.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:11.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:11.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:11.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:11.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:11.176+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:12.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:12.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:12.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:12.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:12.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:12.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:12.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:12.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:12.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:12.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:12.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:12.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:12.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:12.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:12.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:12.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:12.160+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:13.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:13.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:13.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:13.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:13.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:13.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:13.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:13.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:13.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:13.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:13.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:13.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:13.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:13.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:13.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:13.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:13.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:13.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:13.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:13.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:14.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:14.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:14.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:14.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:14.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:14.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:14.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:14.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:14.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:14.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:14.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:14.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:14.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:14.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:14.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:14.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:14.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:14.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:14.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:14.163+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:15.379 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:15.054+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9037 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:15.379 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:15.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:15.379 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:15.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:15.379 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:15.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:15.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:15.205+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:15.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:15.205+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:15.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:15.205+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:15.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:15.205+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:15.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:15.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:15.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:15.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:15.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:15.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:16.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:16.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:16.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:16.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:16.228+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:16.228+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:16.228+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:16.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:16.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:16.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:16.200+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:17.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:17.240+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:17.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:17.240+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:17.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:17.240+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:17.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:17.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:17.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:17.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:17.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:17.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:17.735 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:17.247+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:17.735 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:17.247+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:17.735 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:17.247+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:17.735 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:17.248+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:18.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:18.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:18.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:18.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:18.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:18.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:18.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:18.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:18.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:18.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:18.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:18.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:18.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:18.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:18.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:18.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:18.221+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:19.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:19.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:19.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:19.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:19.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:19.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:19.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:19.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:19.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:19.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:19.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:19.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:19.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:19.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:19.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:19.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:19.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:19.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:19.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:19.182+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:20.385 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:20.056+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9121 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:20.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:20.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:20.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:20.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:20.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:20.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:20.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:20.134+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:20.386 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:20.294+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:20.386 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:20.294+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:20.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:20.294+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:20.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:20.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:20.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:20.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:20.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:20.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:21.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:21.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:21.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:21.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:21.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:21.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:21.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:21.136+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:21.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:21.136+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:21.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:21.136+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:21.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:21.136+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:21.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:21.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:21.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:21.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:21.485 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:21.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:22.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:22.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:22.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:22.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:22.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:22.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:22.103+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:22.103+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:22.103+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:22.103+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:22.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:22.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:22.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:22.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:22.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:22.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:23.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:23.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:23.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:23.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:23.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:23.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:23.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:23.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:23.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:23.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:23.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:23.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:23.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:23.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:23.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:23.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:23.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:23.148+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:24.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:24.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:24.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:24.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:24.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:24.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:24.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:24.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:24.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:24.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:24.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:24.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:24.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:24.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:24.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:24.156+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:25.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:25.058+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9203 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:25.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:25.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:25.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:25.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:25.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:25.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:25.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:25.246+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:25.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:25.246+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:25.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:25.246+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:25.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:25.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:25.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:25.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:25.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:25.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:25.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:25.173+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:26.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:26.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:26.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:26.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:26.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:26.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:26.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:26.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:26.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:26.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:26.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:26.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:26.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:26.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:26.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:26.161+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:27.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:27.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:27.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:27.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:27.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:27.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:27.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:27.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:27.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:27.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:27.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:27.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:27.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:27.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:27.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:27.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:27.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:27.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:27.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:27.113+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:28.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:28.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:28.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:28.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:28.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:28.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:28.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:28.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:28.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:28.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:28.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:28.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:28.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:28.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:28.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:28.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:28.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:28.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:28.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:28.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:29.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:29.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:29.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:29.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:29.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:29.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:29.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:29.110+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:29.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:29.110+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:29.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:29.110+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:29.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:29.111+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:29.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:29.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:29.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:29.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:30.309 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:30.059+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9287 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:30.310 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:30.051+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:30.310 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:30.051+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:30.310 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:30.051+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:30.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:30.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:30.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:30.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:30.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:30.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:30.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:30.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:30.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:30.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:30.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:30.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:30.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:30.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:31.304 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:31.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:31.304 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:31.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:31.304 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:31.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:31.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:31.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:31.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:31.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:31.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:31.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:31.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:31.123+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:31.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:31.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:31.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:31.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:31.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:31.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:32.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:32.011+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:32.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:32.011+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:32.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:32.011+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:32.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:32.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:32.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:32.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:32.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:32.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:32.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:32.153+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:32.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:32.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:32.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:32.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:32.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:32.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:33.305 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:33.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:33.305 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:33.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:33.305 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:33.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:33.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:33.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:33.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:33.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:33.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:33.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:33.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:33.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:33.595 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:33.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:33.595 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:33.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:33.596 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:33.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:34.265 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:33.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:34.265 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:33.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:34.266 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:33.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:34.266 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:34.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:34.266 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:34.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:34.266 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:34.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:34.266 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:34.176+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:34.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:34.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:34.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:34.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:34.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:34.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:35.322 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:35.060+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9369 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:35.322 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:35.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:35.323 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:35.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:35.323 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:35.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:35.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:35.135+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:35.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:35.135+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:35.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:35.135+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:35.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:35.135+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:35.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:35.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:35.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:35.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:35.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:35.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:36.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:36.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:36.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:36.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:36.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:36.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:36.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:36.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:36.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:36.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:36.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:36.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:36.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:36.111+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:36.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:36.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:36.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:36.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:36.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:36.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:37.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:37.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:37.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:37.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:37.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:37.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:37.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:37.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:37.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:37.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:37.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:37.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:37.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:37.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:37.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:37.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:37.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:37.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:37.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:37.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:38.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:38.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:38.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:38.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:38.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:38.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:38.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:38.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:38.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:38.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:38.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:38.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:38.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:38.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:38.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:38.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:38.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:38.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:38.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:38.114+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:39.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:39.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:39.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:39.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:39.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:39.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:39.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:39.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:39.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:39.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:39.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:39.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:39.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:39.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:39.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:39.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:39.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:39.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:39.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:39.162+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:40.409 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:40.062+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9453 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:40.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:40.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:40.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:40.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:40.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:40.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:40.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:40.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:40.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:40.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:40.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:40.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:40.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:40.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:40.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:40.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:40.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:40.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:40.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:40.119+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:41.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:41.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:41.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:41.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:41.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:41.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:41.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:41.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:41.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:41.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:41.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:41.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:41.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:41.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:41.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:41.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:41.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:41.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:41.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:41.152+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:42.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:42.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:42.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:42.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:42.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:42.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:42.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:42.135+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:42.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:42.135+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:42.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:42.135+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:42.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:42.165+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:42.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:42.165+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:42.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:42.165+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:42.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:42.165+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:43.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:43.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:43.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:43.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:43.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:43.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:43.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:43.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:43.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:43.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:43.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:43.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:43.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:43.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:43.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:43.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:43.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:43.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:43.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:43.123+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:44.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:44.145+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:44.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:44.145+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:44.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:44.145+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:44.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:44.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:44.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:44.147+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:44.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:44.065+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:44.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:44.065+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:44.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:44.065+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:45.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:45.063+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9535 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:45.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:45.055+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:45.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:45.055+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:45.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:45.055+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:45.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:45.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:45.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:45.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:45.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:45.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:45.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:45.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:45.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:45.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:45.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:45.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:45.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:45.099+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:46.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:46.063+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:46.063+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:46.063+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:46.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:46.063+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:46.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:46.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:46.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:46.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:46.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:46.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:46.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:46.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:46.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:46.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:46.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:46.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:47.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:47.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:47.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:47.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:47.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:47.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:47.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:47.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:47.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:47.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:47.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:47.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:47.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:47.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:47.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:47.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:47.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:47.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:47.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:47.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:48.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:48.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:48.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:48.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:48.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:48.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:48.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:48.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:48.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:48.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:48.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:48.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:48.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:48.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:48.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:48.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:48.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:48.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:48.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:48.062+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:49.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:49.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:49.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:49.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:49.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:49.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:49.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:49.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:49.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:49.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:49.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:49.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:49.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:49.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:49.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:49.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:49.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:49.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:49.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:49.049+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:50.233 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:50.065+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9618 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:50.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:49.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:50.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:49.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:50.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:49.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:50.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:50.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:50.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:50.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:50.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:50.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:50.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:50.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:50.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:50.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:50.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:50.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:50.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:50.019+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:51.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:51.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:51.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:51.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:51.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:51.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:51.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:50.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:51.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:50.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:51.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:50.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:51.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:51.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:51.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:51.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:51.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:51.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:51.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:51.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:52.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:51.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:52.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:51.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:52.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:51.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:52.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:52.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:52.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:52.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.002+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:52.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:52.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:52.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:52.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:52.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:52.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:52.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:52.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:52.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:52.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:52.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:52.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:53.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:53.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:53.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:53.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:53.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:53.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:53.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:53.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:53.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:53.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:52.964+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:54.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:53.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:54.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:53.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:54.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:53.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:54.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:53.980+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:54.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:53.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:54.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:53.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:54.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:53.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:54.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:54.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:54.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:54.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:54.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:54.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:55.202 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:25:55.067+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9700 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:25:55.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:54.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:55.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:54.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:55.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:54.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:55.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:54.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:55.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:54.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:55.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:54.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:55.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:54.939+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:55.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:55.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:55.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:55.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:55.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:55.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:56.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:55.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:56.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:55.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:56.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:55.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:56.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:55.894+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:56.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:55.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:56.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:55.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:56.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:55.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:56.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:56.294+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:56.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:56.294+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:56.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:56.294+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:57.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:56.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:57.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:56.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:57.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:56.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:57.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:56.869+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:57.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:56.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:57.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:56.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:57.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:56.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:57.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:57.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:57.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:57.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:57.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:57.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:57.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:57.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:57.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:57.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:58.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:57.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:58.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:57.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:58.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:57.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:58.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:58.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:58.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:58.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:58.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:58.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:59.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:59.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:59.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:59.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:59.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:25:59.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:25:59.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:58.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:59.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:58.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:59.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:58.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:25:59.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:58.787+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:59.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:59.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:59.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:59.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:25:59.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:25:59.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:00.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:00.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:00.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:00.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:00.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:00.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:00.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:59.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:00.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:59.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:00.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:59.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:00.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:25:59.789+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:00.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:00.069+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9783 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:00.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:00.190+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:00.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:00.190+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:00.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:00.190+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:01.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:00.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:01.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:00.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:01.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:00.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:01.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:00.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:01.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:01.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:01.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:01.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:01.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:01.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:01.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:01.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:01.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:01.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:01.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:01.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:01.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:01.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:01.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:01.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:01.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:01.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:01.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:01.715+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:02.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:02.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:02.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:02.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:02.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:02.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:02.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:02.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:02.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:02.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:02.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:02.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:03.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:02.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:03.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:02.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:03.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:02.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:03.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:02.754+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:03.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:03.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:03.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:03.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:03.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:03.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:03.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:03.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:03.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:03.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:03.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:03.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:03.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:03.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:03.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:03.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:03.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:03.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:03.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:03.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:04.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:04.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:04.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:04.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:04.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:04.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:04.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:04.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:04.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:04.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:04.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:04.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:05.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:04.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:05.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:04.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:05.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:04.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:05.079 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:04.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:05.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:05.072+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9865 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:05.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:05.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:05.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:05.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:05.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:05.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:05.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:05.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:05.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:05.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:05.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:05.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:06.123 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:05.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:06.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:05.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:06.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:05.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:06.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:05.764+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:06.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:06.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:06.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:06.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:06.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:06.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:06.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:06.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:06.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:06.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:06.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:06.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:07.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:06.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:07.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:06.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:07.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:06.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:07.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:06.804+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:07.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:07.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:07.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:07.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:07.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:07.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:07.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:07.084+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:07.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:07.084+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:07.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:07.084+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:08.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:08.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:08.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:08.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:08.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:08.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:08.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:07.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:08.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:07.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:08.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:07.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:08.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:07.819+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:08.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:08.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:08.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:08.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:08.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:08.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:09.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:09.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:09.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:09.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:09.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:09.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:09.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:08.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:09.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:08.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:09.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:08.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:09.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:08.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:09.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:09.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:09.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:09.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:09.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:09.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:10.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:09.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:10.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:09.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:10.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:09.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:10.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:09.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:10.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:10.074+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 9948 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:10.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:10.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:10.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:10.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:10.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:10.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:10.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:10.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:10.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:10.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:10.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:10.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:11.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:10.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:11.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:10.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:11.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:10.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:11.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:10.805+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:11.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:11.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:11.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:11.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:11.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:11.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:11.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:11.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:11.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:11.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:11.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:11.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:12.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:12.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:12.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:12.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:12.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:12.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:12.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:12.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:12.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:12.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:12.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:12.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:12.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:11.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:12.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:11.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:12.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:11.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:12.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:11.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:13.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:13.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:13.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:13.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:13.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:13.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:13.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:13.026+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:13.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:13.026+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:13.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:13.026+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:13.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:12.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:13.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:12.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:13.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:12.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:13.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:12.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:13.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:13.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:13.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:13.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:13.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:13.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:13.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:13.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:14.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:14.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:14.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:14.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:14.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:14.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:14.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:13.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:14.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:13.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:14.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:13.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:15.081 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:14.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:15.082 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:14.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:15.082 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:14.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:15.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:14.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:15.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:14.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:15.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:14.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:15.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:14.815+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:15.483 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:15.075+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10030 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:15.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:15.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:15.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:15.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:15.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:15.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:16.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:15.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:16.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:15.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:16.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:15.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:16.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:15.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:16.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:16.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:16.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:16.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:16.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:16.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:16.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:15.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:16.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:15.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:16.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:15.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:17.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:17.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:17.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:17.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:17.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:17.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:17.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:16.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:17.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:16.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:17.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:16.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:17.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:16.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:17.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:16.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:17.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:16.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:17.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:16.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:18.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:17.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:18.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:17.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:18.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:17.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:18.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:17.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:18.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:18.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:18.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:18.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:18.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:18.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:18.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:17.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:18.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:17.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:18.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:17.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:19.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:18.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:19.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:18.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:19.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:18.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:19.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:18.886+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:19.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:19.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:19.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:19.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:19.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:19.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:19.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:18.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:19.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:18.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:19.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:18.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:20.166 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:20.077+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10113 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:20.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:20.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:20.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:20.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:20.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:20.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:20.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:19.866+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:20.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:19.866+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:20.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:19.866+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:20.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:19.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:20.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:19.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:20.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:19.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:20.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:19.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:21.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:20.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:21.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:20.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:21.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:20.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:21.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:20.893+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:21.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:20.999+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:21.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:20.999+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:21.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:20.999+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:21.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:20.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:21.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:20.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:21.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:20.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:22.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:22.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:22.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:22.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:22.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:22.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:21.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:21.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:21.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:22.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:21.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:22.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:21.986+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:22.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:21.986+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:22.235 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:21.986+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:23.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:22.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:23.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:22.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:23.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:22.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:23.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:22.980+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:23.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:22.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:23.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:22.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:23.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:22.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:23.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:22.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:23.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:22.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:23.485 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:22.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:24.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:23.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:24.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:23.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:24.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:23.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:24.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:23.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:24.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:23.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:24.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:23.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:24.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:23.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:24.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:23.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:24.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:23.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:24.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:23.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:25.210 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:25.078+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10195 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:25.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:24.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:25.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:24.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:25.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:24.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:25.211 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:24.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:25.211 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:24.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:25.211 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:24.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:25.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:24.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:25.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:24.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:25.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:24.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:25.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:24.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:26.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:25.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:26.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:25.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:26.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:25.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:26.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:25.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:26.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:25.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:26.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:25.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:26.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:25.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:26.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:25.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:26.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:25.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:26.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:25.949+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:27.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:26.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:27.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:26.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:27.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:26.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:27.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:26.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:27.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:26.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:27.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:26.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:27.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:26.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:27.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:26.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:27.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:26.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:27.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:26.908+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:28.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:27.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:28.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:27.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:28.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:27.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:28.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:27.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:28.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:27.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:28.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:27.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:28.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:27.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:28.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:27.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:28.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:27.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:28.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:27.920+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:29.232 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:28.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:29.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:28.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:29.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:28.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:29.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:28.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:29.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:28.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:29.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:28.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:29.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:28.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:29.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:28.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:29.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:28.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:29.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:28.959+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:30.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:30.080+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10278 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:30.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:29.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:30.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:29.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:30.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:29.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:30.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:29.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:30.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:29.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:30.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:29.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:30.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:29.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:30.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:29.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:30.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:29.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:30.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:29.970+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:31.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:30.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:31.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:30.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:31.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:30.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:31.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:30.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:31.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:30.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:31.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:30.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:31.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:30.924+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:31.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:30.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:31.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:30.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:31.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:30.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:32.237 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:32.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:32.237 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:32.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:32.237 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:32.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:32.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:31.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:32.237 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:31.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:32.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:31.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:32.238 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:31.936+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:32.238 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:31.948+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:32.238 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:31.948+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:32.238 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:31.948+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:33.232 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:33.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:33.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:33.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:33.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:33.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:33.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:32.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:33.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:32.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:33.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:32.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:33.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:32.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:33.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:32.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:33.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:32.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:33.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:32.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:34.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:33.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:34.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:33.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:34.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:33.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:34.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:33.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:34.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:33.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:34.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:33.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:34.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:33.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:34.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:34.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:34.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:34.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:34.234 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:34.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:35.218 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:35.082+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10360 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:35.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:35.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:35.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:35.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:35.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:35.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:35.219 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:34.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:35.219 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:34.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:35.219 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:34.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:35.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:35.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:35.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:35.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:35.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:35.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:35.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:35.010+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:35.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:35.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:35.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:35.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:35.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:35.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:36.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:36.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:36.466 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:36.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:36.466 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:36.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:36.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:36.054+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:36.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:36.054+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:36.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:36.054+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:36.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:36.054+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:37.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:36.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:37.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:36.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:37.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:36.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:37.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:37.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:37.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:37.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:37.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:37.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:37.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:37.094+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:37.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:37.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:37.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:37.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:37.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:37.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:38.232 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:38.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:38.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:38.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:38.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:38.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:38.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:38.092+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:38.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:38.092+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:38.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:38.092+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:38.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:38.092+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:38.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:37.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:38.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:37.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:38.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:37.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:39.232 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:39.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:39.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:39.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:39.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:39.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:39.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:39.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:39.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:39.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:39.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:39.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:39.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:39.051+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:39.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:38.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:39.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:38.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:39.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:38.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:40.215 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:40.084+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10443 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:40.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:40.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:40.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:40.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:40.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:40.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:40.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:39.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:40.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:39.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:40.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:39.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:40.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:40.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:40.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:40.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:40.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:40.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:40.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:40.019+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:41.232 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:41.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:41.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:41.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:41.233 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:41.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:41.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:41.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:41.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:41.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:41.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:41.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:41.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:41.039+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:41.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:40.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:41.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:40.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:41.234 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:40.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:42.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:41.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:42.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:41.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:42.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:41.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:42.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:42.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:42.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:42.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:42.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:42.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:42.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:42.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:42.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:42.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:42.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:42.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:42.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:42.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:43.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:42.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:43.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:42.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:43.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:42.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:43.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:43.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:43.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:43.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:43.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:43.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:43.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:43.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:43.599 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:43.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:43.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:43.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:43.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:43.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:44.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:44.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:44.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:44.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:44.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:44.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:44.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:43.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:44.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:43.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:44.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:43.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:44.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:44.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:44.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:44.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:44.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:45.217 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:45.086+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10525 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:45.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:45.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:45.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:45.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:45.218 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:45.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:45.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:45.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:45.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:45.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:45.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:45.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:45.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:45.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:45.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:45.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:45.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:45.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:45.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:45.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:46.280 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:46.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:46.280 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:46.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:46.281 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:46.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:46.281 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:46.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:46.281 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:46.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:46.281 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:46.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:46.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:46.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:46.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:46.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:46.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:46.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:46.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:46.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:47.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:47.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:47.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:47.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:47.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:47.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:47.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:47.078+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:47.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:47.078+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:47.484 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:47.078+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:47.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:47.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:47.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:47.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:47.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:47.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:47.484 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:47.042+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:48.313 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:48.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:48.313 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:48.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:48.313 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:48.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:48.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:48.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:48.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:48.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.033+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:48.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:48.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:48.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:48.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:48.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:48.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:49.347 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:49.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:49.347 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:49.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:49.347 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:49.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:49.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:49.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:49.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:49.348 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:48.995+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:49.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:49.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:49.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:49.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:49.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:49.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:50.378 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:50.089+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10609 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:50.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:49.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:50.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:49.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:50.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:49.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:50.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:50.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:50.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:50.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:50.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:50.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:50.380 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:50.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:50.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:50.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:50.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:50.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:50.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:50.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:51.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:51.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:51.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:51.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:51.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:51.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:51.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:51.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:51.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:51.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:51.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:51.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:51.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:51.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:51.634 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:51.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:52.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:52.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:52.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:52.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:52.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:52.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:52.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:52.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:52.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:52.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:51.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:52.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:52.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:52.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:52.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:52.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:52.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:53.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:53.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:53.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:53.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:53.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:53.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:53.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:52.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:53.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:52.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:53.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:52.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:53.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:52.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:53.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:53.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:53.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:53.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:53.601 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:53.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:54.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:54.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:54.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:54.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:54.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:54.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:54.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:53.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:54.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:53.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:54.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:53.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:54.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:53.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:54.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:54.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:54.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:54.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:54.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:54.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:55.220 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:26:55.091+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10692 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:26:55.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:55.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:55.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:55.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:55.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:55.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:55.221 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:54.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:55.221 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:54.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:55.221 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:54.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:55.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:54.892+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:55.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:55.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:55.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:55.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:55.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:55.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:56.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:56.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:56.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:56.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:56.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:56.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:56.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:55.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:56.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:55.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:56.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:55.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:56.234 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:55.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:56.689 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:56.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:56.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:56.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:56.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:56.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:56.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:56.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:56.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:56.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:56.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:56.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:56.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:56.857+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:57.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:57.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:57.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:57.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:57.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:57.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:57.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:57.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:57.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:57.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:57.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:57.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:58.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:58.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:58.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:58.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:58.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:58.033+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:57.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:57.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:57.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:58.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:57.904+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:58.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:58.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:58.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:58.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:58.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:58.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:59.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:59.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:59.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:59.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:59.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:26:59.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:26:59.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:58.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:59.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:58.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:59.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:58.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:26:59.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:58.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:59.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:59.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:59.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:59.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:26:59.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:26:59.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:00.224 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:00.093+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10776 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:00.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:59.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:00.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:59.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:00.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:59.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:00.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:26:59.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:00.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:00.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:00.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:00.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:00.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:00.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:00.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:00.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:00.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:00.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:00.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:00.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:01.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:01.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:01.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:01.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:01.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:01.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:01.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:00.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:01.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:00.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:01.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:00.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:01.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:00.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:01.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:01.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:01.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:01.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:01.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:01.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:02.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:01.945+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:02.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:01.945+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:02.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:01.945+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:02.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:01.945+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:02.351 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:02.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:02.352 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:02.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:02.352 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:02.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:02.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:02.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:02.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:02.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:02.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:02.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:03.305 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:03.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:03.305 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:03.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:03.305 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:03.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:03.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:02.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:03.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:02.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:03.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:02.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:03.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:02.995+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:03.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:03.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:03.601 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:03.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:03.601 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:03.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:04.340 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:04.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:04.341 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:04.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:04.341 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:04.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:04.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:03.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:04.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:03.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:04.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:03.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:04.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:03.980+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:04.706 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:04.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:04.706 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:04.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:04.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:04.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:05.221 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:05.094+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10859 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:05.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:04.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:05.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:04.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:05.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:04.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:05.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:04.962+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:05.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:05.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:05.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:05.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:05.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:05.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:05.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:05.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:05.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:05.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:05.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:05.295+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:06.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:05.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:06.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:05.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:06.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:05.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:06.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:05.956+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:06.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:06.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:06.420 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:06.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:06.420 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:06.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:06.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:06.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:06.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:06.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:06.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:06.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:07.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:06.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:07.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:06.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:07.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:06.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:07.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:06.916+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:07.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:07.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:07.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:07.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:07.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:07.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:07.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:07.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:07.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:07.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:07.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:07.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:08.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:07.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:08.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:07.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:08.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:07.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:08.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:07.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:08.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:08.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:08.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:08.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:08.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:08.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:08.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:08.187+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:08.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:08.187+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:08.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:08.187+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:09.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:08.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:09.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:08.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:09.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:08.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:09.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:08.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:09.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:09.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:09.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:09.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:09.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:09.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:09.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:09.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:09.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:09.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:09.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:09.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:10.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:10.095+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 10943 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:10.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:09.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:10.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:09.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:10.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:09.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:10.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:09.915+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:10.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:10.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:10.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:10.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:10.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:10.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:10.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:10.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:10.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:10.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:10.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:10.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:11.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:10.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:11.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:10.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:11.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:10.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:11.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:10.917+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:11.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:11.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:11.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:11.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:11.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:11.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:11.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:11.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:11.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:11.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:11.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:11.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:12.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:11.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:12.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:11.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:12.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:11.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:12.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:11.880+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:12.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:12.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:12.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:12.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:12.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:12.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:12.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:12.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:12.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:12.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:12.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:12.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:13.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:12.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:13.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:12.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:13.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:12.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:13.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:12.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:13.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:13.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:13.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:13.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:13.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:13.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:13.602 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:13.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:13.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:13.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:13.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:13.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:14.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:13.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:14.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:13.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:14.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:13.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:14.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:13.906+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:14.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:14.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:14.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:14.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:14.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:14.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:14.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:14.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:14.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:14.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:14.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:14.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:15.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:15.097+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11026 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:14.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:14.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:14.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:14.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:15.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:15.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:15.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:15.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:15.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:15.336+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:15.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:15.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:15.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:15.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:15.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:15.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:15.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:15.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:15.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:15.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:15.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:15.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:15.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:15.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:16.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:16.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:16.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:16.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:16.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:16.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:16.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:16.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:16.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:16.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:16.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:16.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:16.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:16.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:16.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:16.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:16.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:17.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:17.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:17.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:17.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:17.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:17.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:17.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:17.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:17.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:18.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:17.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:18.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:17.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:18.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:17.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:18.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:17.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:18.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:18.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:18.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:18.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:18.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:18.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:18.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:18.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:18.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:18.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:18.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:18.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:19.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:18.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:19.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:18.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:19.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:18.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:19.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:18.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:19.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:19.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:19.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:19.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:19.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:19.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:19.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:19.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:19.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:19.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:19.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:19.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:20.167 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:20.099+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11109 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:20.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:19.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:20.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:19.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:20.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:19.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:20.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:19.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:20.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:20.385+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:20.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:20.385+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:20.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:20.385+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:20.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:20.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:20.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:20.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:20.483 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:20.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:21.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:20.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:21.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:20.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:21.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:20.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:21.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:20.923+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:21.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:21.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:21.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:21.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:21.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:21.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:21.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:21.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:21.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:21.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:21.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:21.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:22.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:21.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:22.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:21.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:22.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:21.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:22.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:21.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:22.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:22.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:22.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:22.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:22.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:22.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:22.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:22.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:22.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:22.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:22.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:22.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:23.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:22.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:23.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:22.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:23.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:22.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:23.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:22.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:23.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:23.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:23.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:23.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:23.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:23.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:23.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:23.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:23.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:23.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:23.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:23.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:24.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:23.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:24.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:23.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:24.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:23.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:24.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:23.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:24.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:24.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:24.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:24.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:24.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:24.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:24.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:24.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:24.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:24.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:24.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:24.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:25.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:24.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:25.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:24.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:25.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:24.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:25.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:24.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:25.437 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:25.101+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11193 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:25.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:25.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:25.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:25.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:25.438 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:25.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:25.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:25.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:25.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:25.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:25.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:25.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:26.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:25.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:26.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:25.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:26.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:25.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:26.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:25.849+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:26.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:26.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:26.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:26.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:26.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:26.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:26.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:26.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:26.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:26.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:26.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:26.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:27.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:26.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:27.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:26.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:27.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:26.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:27.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:26.819+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:27.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:27.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:27.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:27.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:27.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:27.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:27.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:27.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:27.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:27.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:27.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:27.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:28.235 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:27.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:28.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:27.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:28.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:27.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:28.236 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:27.865+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:28.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:28.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:28.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:28.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:28.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:28.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:28.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:28.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:28.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:28.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:28.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:28.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:29.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:28.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:29.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:28.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:29.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:28.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:29.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:28.905+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:29.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:29.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:29.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:29.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:29.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:29.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:29.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:29.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:29.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:29.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:29.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:30.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:30.103+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11276 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:30.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:29.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:30.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:29.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:30.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:29.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:30.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:29.873+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:30.399 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:30.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:30.399 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:30.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:30.400 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:30.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:30.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:30.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:30.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:30.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:30.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:30.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:31.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:30.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:31.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:30.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:31.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:30.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:31.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:30.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:31.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:31.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:31.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:31.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:31.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:31.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:31.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:31.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:31.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:31.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:31.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:31.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:32.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:31.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:32.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:31.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:32.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:31.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:32.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:31.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:32.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:32.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:32.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:32.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:32.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:32.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:32.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:32.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:32.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:32.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:32.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:32.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:33.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:32.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:33.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:32.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:33.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:32.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:33.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:32.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:33.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:33.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:33.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:33.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:33.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:33.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:33.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:33.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:33.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:33.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:33.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:33.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:34.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:33.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:34.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:33.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:34.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:33.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:34.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:33.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:34.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:34.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:34.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:34.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:34.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:34.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:34.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:34.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:34.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:34.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:34.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:34.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:35.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:34.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:35.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:34.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:35.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:34.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:35.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:34.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:35.482 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:35.106+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11360 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:35.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:35.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:35.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:35.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:35.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:35.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:35.767 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:35.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:35.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:35.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:35.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:35.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:36.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:35.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:36.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:35.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:36.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:35.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:36.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:35.763+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:36.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:36.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:36.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:36.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:36.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:36.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:36.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:36.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:36.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:36.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:36.734 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:36.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:37.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:36.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:37.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:36.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:37.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:36.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:37.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:36.729+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:37.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:37.207+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:37.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:37.208+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:37.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:37.208+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:37.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:37.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:37.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:37.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:37.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:37.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:37.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:37.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:37.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:37.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:37.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:37.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:38.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:38.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:38.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:38.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:38.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:38.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:38.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:38.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:38.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:38.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:38.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:38.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:38.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:38.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:38.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:38.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:39.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:39.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:39.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:39.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:39.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:39.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:39.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:39.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:39.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:39.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:39.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:39.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:39.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:39.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:39.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:39.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:39.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:40.457 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:40.108+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11443 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:40.457 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:40.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:40.458 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:40.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:40.458 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:40.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:40.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:40.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:40.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:40.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:40.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:40.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:40.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:40.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:40.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:40.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:40.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:40.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:40.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:40.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:41.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:41.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:41.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:41.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:41.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:41.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:41.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:41.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:41.735 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:41.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:41.736 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:41.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:41.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:41.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:41.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:41.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:41.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:41.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:41.736 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:41.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:42.438 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:42.133+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:42.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:42.133+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:42.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:42.133+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:42.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:42.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:42.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:42.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:42.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:42.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:42.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:42.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:42.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:42.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:42.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:42.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:42.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:42.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:43.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:43.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:43.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:43.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:43.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:43.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:43.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:43.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:43.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:43.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:43.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:43.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:43.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:43.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:43.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:43.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:43.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:43.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:43.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:43.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:44.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:44.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:44.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:44.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:44.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:44.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:44.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:44.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:44.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:44.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:44.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:44.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:44.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:44.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:44.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:44.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:45.416 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:45.109+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11527 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:45.416 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:45.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:45.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:45.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:45.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:45.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:45.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:45.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:45.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:45.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:45.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:45.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:45.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:45.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:45.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:45.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:45.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:45.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:45.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:45.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:46.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:46.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:46.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:46.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:46.482 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:46.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:46.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:46.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:46.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:46.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:46.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:46.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:46.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:46.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:46.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:46.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:47.433 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:47.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:47.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:47.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:47.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:47.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:47.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:47.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:47.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:47.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:47.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:47.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:47.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:47.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:47.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:47.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:47.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:47.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:47.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:47.556+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:48.595 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:48.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:48.595 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:48.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:48.595 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:48.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:48.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:48.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:48.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:48.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:48.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:48.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:48.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:48.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:48.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:48.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:48.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:48.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:48.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:48.590+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:49.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:49.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:49.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:49.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:49.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:49.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:49.597 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:49.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:49.597 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:49.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:49.597 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:49.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:49.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:49.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:49.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:49.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:49.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:49.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:49.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:49.592+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:50.460 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:50.112+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11610 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:50.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:50.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:50.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:50.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:50.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:50.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:50.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:50.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:50.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:50.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:50.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:50.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:50.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:50.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:50.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:50.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:51.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:51.230+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:51.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:51.230+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:51.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:51.230+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:51.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:51.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:51.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:51.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:51.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:51.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:51.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:51.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:51.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:51.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:51.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:51.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:51.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:51.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:52.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:52.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:52.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:52.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:52.483 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:52.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:52.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:52.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:52.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:52.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:52.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:52.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:52.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:52.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:52.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:52.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:52.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:52.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:52.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:52.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:53.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:53.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:53.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:53.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:53.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:53.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:53.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:53.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:53.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:53.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:53.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:53.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:53.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:53.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:53.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:53.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:53.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:53.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:53.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:53.635+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:54.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:54.249+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:54.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:54.249+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:54.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:54.249+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:54.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:54.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:54.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:54.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:54.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:54.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:54.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:54.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:54.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:54.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:54.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:54.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:54.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:54.603+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:55.482 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:27:55.112+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11693 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:27:55.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:55.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:55.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:55.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:55.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:55.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:55.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:55.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:55.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:55.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:55.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:55.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:55.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:55.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:55.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:55.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:55.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:55.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:55.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:55.564+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:56.547 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:56.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:56.547 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:56.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:56.547 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:56.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:56.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:56.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:56.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:56.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:56.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:56.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:56.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:56.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:56.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:56.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:56.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:56.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:56.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:56.541+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:57.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:57.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:57.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:57.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:57.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:57.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:57.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:57.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:57.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:57.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:57.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:57.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:57.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:57.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:57.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:57.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:57.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:57.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:57.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:57.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:58.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:58.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:58.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:58.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:58.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:58.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:58.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:58.537+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:58.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:58.537+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:58.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:58.537+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:58.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:58.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:58.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:58.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:58.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:58.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:58.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:59.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:59.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:59.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:59.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:59.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:27:59.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:27:59.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:59.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:59.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:59.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:59.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:27:59.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:27:59.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:59.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:59.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:59.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:59.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:59.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:27:59.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:27:59.529+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:00.482 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:00.115+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11775 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:00.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:00.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:00.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:00.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:00.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:00.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:00.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:00.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:00.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:00.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:00.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:00.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:00.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:00.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:00.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:00.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:00.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:00.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:00.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:00.504+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:01.602 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:01.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:01.602 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:01.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:01.602 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:01.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:01.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:01.485+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:01.603 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:01.485+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:01.603 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:01.485+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:01.603 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:01.485+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:01.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:01.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:01.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:01.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:01.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:01.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:02.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:02.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:02.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:02.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:02.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:02.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:02.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:02.580+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:02.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:02.580+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:02.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:02.580+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:02.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:02.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:02.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:02.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:02.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:02.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:02.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:02.487+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:03.601 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:03.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:03.601 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:03.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:03.602 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:03.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:03.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:03.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:03.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:03.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:03.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:03.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:03.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:03.503+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:03.596+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:03.596+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:03.596+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:04.156 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-19T19:28:04.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T19:28:04.162 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-19T19:28:04.162 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T19:28:04.163 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-02-19T19:28:04.183 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-19T19:28:04.194 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task sleep... 2024-02-19T19:28:04.195 INFO:teuthology.task.sleep:Sleeping for 3600 seconds 2024-02-19T19:28:04.623 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:04.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:04.624 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:04.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:04.624 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:04.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:04.624 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:04.474+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:04.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:04.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:04.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:04.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:04.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:04.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:04.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:04.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:04.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:04.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:04.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:04.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:05.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:05.116+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11858 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:05.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:05.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:05.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:05.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:05.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:05.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:05.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:05.437+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:05.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:05.437+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:05.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:05.437+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:05.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:05.437+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:05.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:05.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:05.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:05.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:05.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:05.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:06.633 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:06.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:06.634 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:06.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:06.634 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:06.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:06.634 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:06.453+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:06.634 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:06.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:06.634 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:06.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:06.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:06.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:06.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:06.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:06.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:06.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:06.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:06.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:07.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:07.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:07.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:07.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:07.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:07.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:07.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:07.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:07.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:07.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:07.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:07.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:07.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:07.476+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:07.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:07.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:07.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:07.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:07.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:08.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:08.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:08.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:08.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:08.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:08.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:08.676 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:08.440+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:08.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:08.440+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:08.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:08.440+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:08.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:08.440+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:08.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:08.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:08.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:08.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:09.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:09.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:09.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:09.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:09.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:09.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:09.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:09.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:09.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:09.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:09.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:09.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:09.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:09.424+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:09.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:09.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:09.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:09.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:09.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:09.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:10.393 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:10.118+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 11940 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:10.393 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:10.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:10.394 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:10.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:10.394 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:10.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:10.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:10.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:10.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:10.388+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:10.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:10.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:10.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:10.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:10.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:10.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:11.682 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:11.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:11.682 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:11.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:11.682 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:11.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:11.682 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:11.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:11.682 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:11.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:11.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:11.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:11.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:11.430+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:11.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:11.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:11.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:11.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:11.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:11.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:12.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:12.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:12.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:12.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:12.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:12.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:12.659 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:12.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:12.659 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:12.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:12.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:12.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:12.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:12.406+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:12.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:12.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:12.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:12.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:12.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:13.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:13.322+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:13.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:13.322+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:13.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:13.322+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:13.606 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:13.361+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:13.606 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:13.361+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:13.606 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:13.361+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:13.606 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:13.361+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:13.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:13.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:13.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:13.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:13.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:13.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:14.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:14.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:14.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:14.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:14.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:14.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:14.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:14.396+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:14.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:14.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:14.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:14.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:14.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:14.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:14.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:14.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:14.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:14.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:14.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:14.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:15.410 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:15.121+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12023 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:15.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:15.436+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:15.436+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:15.436+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:15.436+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:15.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:15.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:15.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:15.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:15.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:15.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:15.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:15.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:15.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:15.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:15.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:15.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:16.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:16.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:16.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:16.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:16.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:16.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:16.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:16.408+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:16.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:16.599+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:16.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:16.599+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:16.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:16.599+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:16.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:16.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:16.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:16.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:16.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:16.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:17.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:17.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:17.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:17.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:17.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:17.389+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:17.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:17.389+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:17.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:17.389+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:17.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:17.389+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:17.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:17.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:17.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:17.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:17.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:17.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:18.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:18.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:18.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:18.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:18.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:18.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:18.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:18.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:18.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:18.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:18.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:18.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:18.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:18.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:18.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:18.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:18.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:18.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:18.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:18.386+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:19.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:19.414+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:19.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:19.414+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:19.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:19.414+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:19.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:19.414+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:19.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:19.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:19.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:19.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:19.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:19.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:19.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:19.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:19.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:19.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:19.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:19.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:20.424 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:20.123+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12105 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:20.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:20.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:20.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:20.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:20.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:20.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:20.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:20.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:20.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:20.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:20.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:20.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:20.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:20.420+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:20.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:20.420+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:20.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:20.420+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:20.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:20.420+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:21.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:21.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:21.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:21.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:21.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:21.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:21.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:21.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:21.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:21.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:21.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:21.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:21.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:21.419+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:21.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:21.419+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:21.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:21.419+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:21.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:21.419+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:22.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:22.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:22.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:22.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:22.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:22.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:22.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:22.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:22.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:22.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:22.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:22.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:22.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:22.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:22.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:22.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:22.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:22.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:22.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:22.458+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:23.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:23.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:23.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:23.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:23.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:23.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:23.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:23.441+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:23.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:23.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:23.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:23.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:23.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:23.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:23.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:23.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:23.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:23.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:23.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:23.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:24.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:24.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:24.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:24.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:24.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:24.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:24.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:24.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:24.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:24.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:24.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:24.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:24.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:24.464+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:24.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:24.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:24.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:24.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:24.734 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:24.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:25.399 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:25.126+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12188 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:25.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:25.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:25.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:25.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:25.658 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:25.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:25.658 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:25.448+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:25.658 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:25.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:25.658 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:25.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:25.658 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:25.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:25.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:25.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:25.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:25.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:25.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:25.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:26.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:26.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:26.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:26.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:26.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:26.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:26.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:26.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:26.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:26.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:26.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:26.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:26.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:26.430+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:26.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:26.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:26.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:26.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:26.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:27.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:27.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:27.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:27.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:27.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:27.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:27.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:27.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:27.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:27.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:27.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:27.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:27.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:27.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:27.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:27.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:27.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:27.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:28.659 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:28.416+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:28.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:28.416+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:28.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:28.416+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:28.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:28.416+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:28.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:28.339+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:28.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:28.339+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:28.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:28.339+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:28.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:28.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:28.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:28.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:28.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:28.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:29.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:29.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:29.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:29.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:29.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:29.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:29.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:29.372+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:29.622 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:29.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:29.622 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:29.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:29.622 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:29.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:29.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:29.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:29.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:29.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:30.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:30.128+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12270 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:30.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:30.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:30.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:30.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:30.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:30.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:30.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:30.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:30.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:30.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:30.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:30.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:30.483 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:30.351+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:30.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:30.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:30.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:30.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:31.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:31.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:31.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:31.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:31.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:31.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:31.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:31.367+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:31.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:31.367+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:31.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:31.367+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:31.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:31.368+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:31.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:31.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:31.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:31.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:31.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:31.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:32.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:32.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:32.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:32.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:32.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:32.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:32.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:32.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:32.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:32.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:32.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:32.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:32.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:32.352+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:32.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:32.352+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:32.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:32.352+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:32.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:32.352+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:33.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:33.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:33.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:33.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:33.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:33.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:33.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:33.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:33.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:33.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:33.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:33.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:33.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:33.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:33.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:33.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:33.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:33.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:33.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:33.396+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:34.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:34.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:34.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:34.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:34.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:34.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:34.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:34.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:34.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:34.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:34.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:34.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:34.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:34.397+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:34.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:34.397+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:34.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:34.397+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:34.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:34.398+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:35.444 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:35.130+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12353 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:35.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:35.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:35.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:35.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:35.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:35.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:35.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:35.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:35.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:35.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:35.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:35.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:35.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:35.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:35.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:35.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:35.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:35.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:35.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:35.440+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:36.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:36.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:36.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:36.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:36.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:36.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:36.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:36.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:36.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:36.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:36.733 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:36.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:36.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:36.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:36.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:36.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:36.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:36.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:36.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:36.433+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:37.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:37.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:37.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:37.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:37.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:37.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:37.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:37.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:37.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:37.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:37.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:37.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:37.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:37.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:37.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:37.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:37.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:37.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:37.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:37.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:38.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:38.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:38.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:38.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:38.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:38.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:38.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:38.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:38.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:38.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:38.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:38.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:38.463+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:38.463+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:38.463+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:38.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:38.463+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:39.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:39.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:39.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:39.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:39.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:39.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:39.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:39.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:39.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:39.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:39.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:39.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:39.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:39.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:39.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:39.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:39.489+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:40.406 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:40.132+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12435 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:40.407 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:40.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:40.407 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:40.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:40.407 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:40.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:40.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:40.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:40.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:40.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:40.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:40.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:40.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:40.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:40.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:40.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:40.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:40.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:40.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:40.540+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:41.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:41.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:41.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:41.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:41.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:41.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:41.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:41.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:41.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:41.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:41.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:41.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:41.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:41.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:41.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:41.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:41.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:41.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:41.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:41.541+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:42.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:42.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:42.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:42.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:42.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:42.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:42.597 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:42.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:42.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:42.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:42.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:42.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:42.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:42.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:42.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:42.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:42.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:42.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:42.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:42.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:43.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:43.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:43.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:43.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:43.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:43.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:43.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:43.511+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:43.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:43.511+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:43.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:43.511+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:43.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:43.511+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:43.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:43.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:43.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:43.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:43.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:43.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:44.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:44.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:44.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:44.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:44.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:44.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:44.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:44.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:44.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:44.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:44.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:44.490+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:45.435 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:45.133+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12518 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:45.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:45.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:45.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:45.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:45.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:45.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:45.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:45.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:45.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:45.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:45.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:45.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:45.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:45.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:45.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:45.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:45.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:45.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:45.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:45.474+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:46.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:46.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:46.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:46.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:46.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:46.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:46.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:46.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:46.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:46.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:46.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:46.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:46.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:46.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:46.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:46.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:46.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:46.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:46.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:46.490+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:47.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:47.536+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:47.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:47.536+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:47.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:47.536+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:47.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:47.536+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:47.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:47.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:47.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:47.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:47.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:47.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:47.640 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:47.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:47.640 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:47.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:47.640 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:47.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:48.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:48.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:48.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:48.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:48.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:48.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:48.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:48.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:48.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:48.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:48.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:48.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:48.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:48.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:48.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:48.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:48.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:48.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:48.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:48.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:49.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:49.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:49.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:49.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:49.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:49.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:49.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:49.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:49.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:49.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:49.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:49.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:49.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:49.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:49.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:49.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:49.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:49.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:49.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:49.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:50.465 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:50.135+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12600 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:50.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:50.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:50.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:50.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:50.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:50.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:50.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:50.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:50.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:50.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:50.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:50.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:50.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:50.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:50.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:50.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:50.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:51.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:51.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:51.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:51.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:51.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:51.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:51.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:51.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:51.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:51.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:51.539+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:51.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:51.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:51.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:51.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:51.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:51.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:52.681 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:52.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:52.681 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:52.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:52.682 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:52.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:52.682 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:52.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:52.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:52.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:52.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:52.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:52.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:52.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:52.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:52.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:52.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:52.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:52.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:52.496+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:53.608 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:53.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:53.608 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:53.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:53.609 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:53.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:53.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:53.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:53.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:53.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:53.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:53.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:53.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:53.508+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:53.610 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:53.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:53.610 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:53.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:53.610 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:53.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:54.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:54.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:54.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:54.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:54.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:54.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:54.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:54.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:54.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:54.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:54.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:54.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:54.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:54.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:54.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:54.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:54.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:54.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:54.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:54.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:55.415 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:28:55.137+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12683 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:28:55.415 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:55.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:55.415 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:55.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:55.415 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:55.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:55.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:55.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:55.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:55.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:55.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:55.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:55.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:55.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:55.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:55.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:55.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:55.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:55.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:55.487+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:56.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:56.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:56.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:56.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:56.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:56.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:56.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:56.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:56.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:56.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:56.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:56.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:56.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:56.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:56.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:56.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:56.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:56.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:56.509+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:57.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:57.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:57.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:57.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:57.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:57.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:57.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:57.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:57.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:57.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:57.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:57.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:57.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:57.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:57.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:57.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:57.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:57.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:57.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:57.467+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:58.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:58.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:58.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:58.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:58.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:58.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:58.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:58.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:58.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:58.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:58.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:58.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:58.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:58.440+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:58.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:58.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:58.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:58.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:58.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:58.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:59.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:59.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:59.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:59.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:59.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:28:59.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:28:59.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:59.421+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:59.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:59.421+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:59.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:59.421+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:28:59.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:28:59.421+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:59.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:59.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:59.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:59.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:28:59.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:28:59.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:00.455 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:00.139+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12767 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:00.455 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:00.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:00.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:00.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:00.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:00.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:00.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:00.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:00.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:00.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:00.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:00.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:00.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:00.450+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:00.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:00.450+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:00.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:00.450+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:00.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:00.450+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:01.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:01.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:01.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:01.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:01.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:01.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:01.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:01.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:01.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:01.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:01.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:01.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:01.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:01.482+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:01.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:01.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:01.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:01.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:01.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:01.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:02.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:02.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:02.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:02.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:02.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:02.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:02.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:02.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:02.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:02.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:02.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:02.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:02.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:02.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:02.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:02.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:02.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:02.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:02.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:02.508+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:03.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:03.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:03.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:03.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:03.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:03.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:03.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:03.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:03.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:03.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:03.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:03.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:03.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:03.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:03.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:04.585 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:04.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:04.585 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:04.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:04.585 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:04.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:04.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:04.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:04.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:04.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:04.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:04.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:04.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:04.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:04.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:04.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:05.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:05.141+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12848 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:05.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:05.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:05.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:05.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:05.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:05.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:05.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:05.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:05.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:05.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:05.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:05.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:05.545+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:05.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:05.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:05.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:05.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:05.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:05.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:06.528 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:06.348+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:06.529 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:06.348+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:06.529 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:06.348+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:06.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:06.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:06.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:06.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:06.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:06.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:06.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:06.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:06.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:06.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:06.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:06.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:06.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:06.524+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:07.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:07.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:07.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:07.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:07.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:07.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:07.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:07.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:07.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:07.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:07.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:07.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:07.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:07.543+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:07.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:07.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:07.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:08.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:08.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:08.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:08.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:08.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:08.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:08.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:08.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:08.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:08.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:08.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:08.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:08.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:08.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:08.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:08.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:08.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:09.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:09.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:09.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:09.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:09.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:09.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:09.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:09.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:09.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:09.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:09.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:09.503+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:09.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:09.503+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:09.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:09.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:09.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:09.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:09.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:09.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:10.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:10.144+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12919 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:10.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:10.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:10.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:10.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:10.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:10.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:10.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:10.493+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:10.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:10.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:10.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:10.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:10.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:10.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:11.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:11.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:11.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:11.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:11.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:11.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:11.701 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:11.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:11.701 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:11.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:11.701 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:11.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:11.701 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:11.494+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:11.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:11.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:11.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:11.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:11.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:11.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:12.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:12.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:12.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:12.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:12.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:12.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:12.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:12.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:12.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:12.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:12.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:12.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:12.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:12.527+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:12.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:12.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:12.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:13.610 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:13.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:13.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:13.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:13.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:13.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:13.611 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:13.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:13.611 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:13.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:13.611 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:13.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:13.611 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:13.523+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:13.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:13.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:13.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:13.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:13.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:13.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:14.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:14.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:14.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:14.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:14.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:14.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:14.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:14.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:14.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:14.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:14.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:14.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:14.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:14.499+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:14.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:14.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:14.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:14.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:14.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:14.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:15.447 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:15.146+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 12990 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:15.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:15.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:15.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:15.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:15.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:15.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:15.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:15.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:15.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:15.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:15.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:15.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:15.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:15.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:15.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:15.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:15.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:16.657 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:16.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:16.657 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:16.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:16.657 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:16.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:16.658 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:16.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:16.658 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:16.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:16.658 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:16.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:16.658 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:16.556+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:16.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:16.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:16.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:16.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:16.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:16.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:17.549 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:17.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:17.549 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:17.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:17.549 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:17.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:17.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:17.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:17.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:17.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:17.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:17.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:17.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:17.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:17.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:17.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:17.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:17.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:17.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:17.545+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:18.580 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:18.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:18.580 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:18.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:18.580 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:18.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:18.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:18.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:18.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:18.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:18.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:18.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:18.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:18.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:18.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:18.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:18.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:18.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:18.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:18.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:19.621 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:19.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:19.621 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:19.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:19.621 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:19.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:19.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:19.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:19.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:19.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:19.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:19.682+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:19.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:19.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:19.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:19.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:19.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:19.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:19.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:19.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:20.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:20.149+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13061 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:20.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:20.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:20.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:20.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:20.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:20.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:20.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:20.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:20.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:20.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:20.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:20.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:20.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:20.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:20.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:20.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:20.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:20.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:20.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:20.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:21.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:21.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:21.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:21.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:21.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:21.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:21.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:21.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:21.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:21.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:21.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:21.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:21.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:21.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:21.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:21.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:21.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:21.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:21.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:21.630+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:22.615 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:22.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:22.615 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:22.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:22.615 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:22.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:22.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:22.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:22.935 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:22.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:22.935 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:22.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:22.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:22.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:22.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:22.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:22.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:22.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:22.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:22.611+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:23.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:23.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:23.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:23.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:23.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:23.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:23.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:23.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:23.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:23.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:23.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:23.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:23.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:23.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:23.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:23.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:23.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:23.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:23.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:23.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:24.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:24.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:24.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:24.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:24.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:24.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:24.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:24.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:24.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:24.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:24.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:24.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:24.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:24.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:24.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:24.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:24.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:24.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:25.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:25.151+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13132 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:25.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:25.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:25.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:25.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:25.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:25.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:25.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:25.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:25.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:25.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:25.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:25.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:25.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:25.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:25.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:25.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:25.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:25.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:25.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:25.665+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:26.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:26.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:26.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:26.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:26.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:26.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:26.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:26.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:26.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:26.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:26.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:26.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:26.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:27.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:27.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:27.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:27.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:27.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:27.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:27.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:27.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:27.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:27.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:27.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:27.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:27.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:27.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:27.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:28.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:28.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:28.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:28.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:28.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:28.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:28.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:28.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:28.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:28.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:28.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:28.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:28.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:28.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:28.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:28.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:28.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:28.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:28.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:28.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:29.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:29.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:29.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:29.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:29.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:29.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:29.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:29.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:29.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:29.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:29.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:29.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:29.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:29.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:29.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:29.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:29.708+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:30.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:30.152+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13203 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:30.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:30.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:30.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:30.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:30.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:30.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:30.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:30.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:30.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:30.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:30.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:30.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:30.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:30.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:30.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:30.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:30.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:31.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:31.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:31.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:31.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:31.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:31.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:31.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:31.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:31.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:31.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:31.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:31.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:31.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:31.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:31.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:31.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:31.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:32.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:32.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:32.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:32.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:32.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:32.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:32.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:32.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:32.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:32.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:32.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:32.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:32.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:32.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:32.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:32.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:32.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:32.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:32.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:32.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:33.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:33.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:33.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:33.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:33.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:33.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:33.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:33.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:33.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:33.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:33.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:33.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:33.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:33.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:33.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:33.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:33.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:33.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:33.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:33.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:34.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:34.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:34.606 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:34.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:34.606 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:34.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:34.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:34.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:34.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:34.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:34.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:34.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:34.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:34.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:34.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:34.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:34.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:35.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:35.154+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13274 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:35.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:35.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:35.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:35.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:35.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:35.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:35.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:35.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:35.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:35.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:35.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:35.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:35.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:35.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:35.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:35.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:36.667 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:36.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:36.667 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:36.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:36.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:36.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:36.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:36.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:36.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:36.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:36.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:36.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:36.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:36.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:36.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:36.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:36.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:36.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:36.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:36.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:37.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:37.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:37.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:37.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:37.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:37.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:37.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:37.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:37.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:37.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:37.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:37.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:37.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:37.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:37.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:37.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:37.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:38.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:38.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:38.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:38.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:38.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:38.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:38.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:38.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:38.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:38.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:38.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:38.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:38.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:38.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:38.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:38.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:39.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:39.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:39.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:39.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:39.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:39.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:39.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:39.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:39.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:39.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:39.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:39.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:39.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:40.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:40.156+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13345 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:40.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:40.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:40.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:40.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:40.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:40.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:40.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:40.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:40.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:40.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:40.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:40.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:40.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:40.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:40.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:40.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:40.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:41.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:41.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:41.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:41.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:41.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:41.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:41.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:41.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:41.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:41.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:41.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:41.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:41.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:41.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:41.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:41.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:41.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:41.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:41.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:42.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:42.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:42.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:42.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:42.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:42.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:42.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:42.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:42.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:42.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:42.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:42.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:42.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:42.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:42.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:42.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:42.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:42.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:42.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:43.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:43.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:43.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:43.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:43.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:43.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:43.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:43.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:43.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:43.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:43.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:43.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:43.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:43.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:43.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:43.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:43.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:43.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:43.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:43.683+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:44.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:44.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:44.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:44.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:44.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:44.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:44.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:44.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:44.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:44.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:44.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:44.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:44.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:44.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:44.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:44.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:44.635+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:45.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:45.158+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13432 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:45.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:45.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:45.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:45.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:45.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:45.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:45.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:45.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:45.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:45.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:45.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:45.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:45.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:45.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:45.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:45.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:45.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:45.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:45.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:45.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:46.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:46.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:46.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:46.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:46.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:46.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:46.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:46.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:46.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:46.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:46.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:46.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:46.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:46.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:46.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:46.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:46.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:47.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:47.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:47.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:47.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:47.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:47.323+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:47.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:47.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:47.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:47.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:47.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:47.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:47.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:47.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:47.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:47.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:47.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:47.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:48.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:48.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:48.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:48.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:48.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:48.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:48.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:48.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:48.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:48.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:48.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:48.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:48.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:48.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:48.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:48.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:48.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:48.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:48.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:48.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:49.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:49.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:49.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:49.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:49.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:49.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:49.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:49.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:49.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:49.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:49.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:49.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:49.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:49.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:49.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:49.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:49.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:49.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:49.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:49.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:50.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:50.161+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13503 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:50.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:50.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:50.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:50.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:50.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:50.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:50.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:50.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:50.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:50.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:50.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:50.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:50.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:50.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:50.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:50.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:50.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:50.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:50.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:50.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:51.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:51.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:51.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:51.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:51.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:51.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:51.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:51.586+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:51.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:51.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:51.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:51.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:51.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:51.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:51.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:51.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:51.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:51.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:51.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:51.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:52.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:52.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:52.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:52.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:52.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:52.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:52.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:52.556+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:52.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:52.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:52.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:52.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:52.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:52.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:52.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:52.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:52.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:52.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:52.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:52.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:53.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:53.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:53.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:53.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:53.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:53.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:53.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:53.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:53.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:53.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:53.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:53.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:53.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:53.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:53.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:53.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:53.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:53.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:53.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:53.585+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:54.699 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:54.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:54.699 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:54.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:54.699 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:54.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:54.700 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:54.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:54.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:54.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:54.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:54.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:54.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:54.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:54.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:54.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:54.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:54.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:54.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:54.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:55.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:29:55.164+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13574 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:29:55.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:55.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:55.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:55.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:55.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:55.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:55.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:55.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:55.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:55.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:55.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:55.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:55.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:55.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:55.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:55.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:55.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:55.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:55.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:55.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:56.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:56.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:56.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:56.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:56.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:56.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:56.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:56.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:56.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:56.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:56.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:56.582+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:57.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:56.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:57.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:56.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:57.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:56.731+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:57.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:57.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:57.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:57.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:57.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:57.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:57.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:57.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:57.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:57.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:57.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:57.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:57.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:57.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:57.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:57.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:57.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:58.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:58.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:58.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:58.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:58.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:58.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:58.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:58.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:58.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:58.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:58.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:58.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:58.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:58.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:58.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:58.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:58.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:59.577 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:59.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:59.577 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:59.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:59.578 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:29:59.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:29:59.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:59.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:59.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:59.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:59.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:29:59.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:29:59.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:59.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:59.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:59.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:59.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:59.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:29:59.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:29:59.574+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:00.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:00.166+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13645 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:00.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:00.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:00.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:00.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:00.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:00.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:00.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:00.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:00.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:00.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:00.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:00.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:00.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:00.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:00.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:00.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:00.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:00.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:00.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:00.551+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:01.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:01.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:01.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:01.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:01.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:01.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:01.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:01.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:01.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:01.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:01.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:01.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:01.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:01.572+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:01.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:01.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:01.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:01.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:01.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:01.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:02.697 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:02.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:02.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:02.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:02.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:02.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:02.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:02.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:02.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:02.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:02.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:02.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:02.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:02.551+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:02.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:02.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:02.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:02.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:02.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:02.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:03.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:03.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:03.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:03.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:03.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:03.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:03.553+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:03.553+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:03.553+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:03.553+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:03.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:03.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:03.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:03.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:03.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:04.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:04.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:04.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:04.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:04.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:04.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:04.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:04.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:04.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:04.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:04.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:04.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:04.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:04.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:05.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:04.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:05.173 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:04.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:05.173 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:04.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:05.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:05.168+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13716 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:05.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:05.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:05.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:05.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:05.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:05.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:05.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:05.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:05.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:05.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:05.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:05.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:05.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:05.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:05.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:05.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:05.513+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:06.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:06.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:06.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:06.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:06.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:06.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:06.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:06.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:06.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:06.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:06.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:06.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:06.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:06.494+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:06.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:06.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:06.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:06.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:06.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:06.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:07.651 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:07.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:07.651 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:07.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:07.651 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:07.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:07.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:07.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:07.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:07.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:07.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:07.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:07.652 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:07.453+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:07.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:07.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:07.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:07.647+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:08.623 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:08.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:08.623 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:08.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:08.623 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:08.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:08.623 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:08.491+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:08.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:08.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:08.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:08.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:08.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:08.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:08.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:08.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:08.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:08.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:09.638 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:09.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:09.638 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:09.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:09.638 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:09.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:09.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:09.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:09.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:09.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:09.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:09.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:09.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:09.466+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:09.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:09.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:09.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:09.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:09.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:10.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:10.170+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13787 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:10.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:10.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:10.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:10.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:10.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:10.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:10.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:10.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:10.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:10.512+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:10.512+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:10.512+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:10.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:10.512+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:11.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:11.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:11.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:11.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:11.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:11.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:11.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:11.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:11.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:11.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:11.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:11.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:11.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:11.498+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:11.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:11.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:11.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:11.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:11.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:11.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:12.627 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:12.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:12.627 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:12.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:12.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:12.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:12.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:12.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:12.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:12.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:12.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:12.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:12.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:12.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:12.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:12.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:12.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:12.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:13.615 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:13.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:13.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:13.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:13.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:13.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:13.616 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:13.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:13.616 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:13.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:13.617 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:13.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:13.617 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:13.521+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:13.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:13.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:13.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:13.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:13.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:13.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:14.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:14.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:14.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:14.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:14.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:14.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:14.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:14.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:14.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:14.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:14.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:14.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:14.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:14.506+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:14.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:14.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:14.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:14.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:14.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:15.481 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:15.172+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13857 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:15.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:15.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:15.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:15.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:15.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:15.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:15.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:15.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:15.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:15.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:15.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:15.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:15.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:15.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:15.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:15.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:15.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:15.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:15.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:15.545+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:16.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:16.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:16.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:16.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:16.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:16.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:16.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:16.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:16.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:16.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:16.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:16.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:16.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:16.589+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:16.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:16.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:16.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:16.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:16.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:16.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:17.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:17.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:17.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:17.577+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:17.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:17.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:17.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:17.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:17.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:17.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:17.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:17.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:17.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:17.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:17.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:18.679 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:18.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:18.679 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:18.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:18.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:18.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:18.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:18.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:18.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:18.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:18.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:18.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:18.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:18.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:18.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:18.674+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:18.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:18.674+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:18.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:18.674+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:19.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:19.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:19.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:19.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:19.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:19.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:19.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:19.548+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:19.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:19.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:19.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:19.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:19.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:19.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:19.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:19.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:19.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:19.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:19.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:19.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:20.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:20.174+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13929 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:20.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:20.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:20.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:20.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:20.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:20.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:20.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:20.573+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:20.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:20.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:20.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:20.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:20.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:20.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:20.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:20.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:20.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:20.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:20.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:20.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:21.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:21.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:21.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:21.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:21.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:21.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:21.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:21.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:21.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:21.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:21.535+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:21.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:21.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:21.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:21.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:21.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:21.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:22.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:22.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:22.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:22.498+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:22.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:22.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:22.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:22.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:22.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:22.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:22.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:22.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:22.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:22.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:22.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:22.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:23.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:23.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:23.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:23.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:23.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:23.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:23.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:23.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:23.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:23.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:23.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:23.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:23.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:23.530+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:23.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:23.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:23.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:23.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:23.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:23.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:24.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:24.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:24.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:24.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:24.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:24.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:24.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:24.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:24.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:24.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:24.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:24.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:24.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:24.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:24.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:24.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:24.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:24.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:24.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:25.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:25.176+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 13999 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:25.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:25.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:25.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:25.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:25.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:25.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:25.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:25.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:25.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:25.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:25.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:25.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:25.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:25.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:25.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:26.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:26.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:26.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:26.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:26.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:26.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:26.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:26.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:26.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:26.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:26.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:26.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:26.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:26.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:27.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:27.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:27.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:27.593+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:27.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:27.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:27.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:27.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:27.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:27.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:27.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:27.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:27.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:28.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:28.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:28.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:28.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:28.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:28.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:28.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:28.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:28.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:28.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:28.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:28.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:28.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:28.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:28.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:28.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:28.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:29.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:29.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:29.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:29.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:29.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:29.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:29.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:29.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:29.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:29.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:29.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:29.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:29.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:29.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:29.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:29.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:29.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:29.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:29.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:29.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:30.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:30.178+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14071 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:30.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:30.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:30.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:30.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:30.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:30.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:30.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:30.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:30.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:30.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:30.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:30.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:30.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:30.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:30.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:30.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:30.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:30.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:31.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:31.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:31.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:31.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:31.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:31.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:31.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:31.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:31.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:31.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:31.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:31.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:31.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:31.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:32.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:32.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:32.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:32.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:32.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:32.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:32.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:32.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:32.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:32.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:32.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:32.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:32.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:32.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:32.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:32.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:32.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:32.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:32.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:33.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:33.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:33.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:33.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:33.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:33.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:33.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:33.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:33.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:33.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:33.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:33.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:33.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:33.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:33.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:33.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:33.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:33.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:33.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:34.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:34.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:34.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:34.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:34.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:34.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:34.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:34.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:34.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:34.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:34.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:34.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:34.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:34.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:34.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:34.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:34.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:35.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:35.181+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14141 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:35.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:35.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:35.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:35.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:35.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:35.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:35.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:35.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:35.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:35.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:35.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:35.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:35.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:35.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:35.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:35.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:35.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:36.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:36.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:36.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:36.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:36.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:36.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:36.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:36.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:36.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:36.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:36.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:36.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:36.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:36.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:36.817 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:36.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:36.817 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:36.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:36.817 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:36.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:37.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:37.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:37.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:37.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:37.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:37.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:37.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:37.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:37.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:37.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:37.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:37.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:37.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:37.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:37.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:37.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:37.675+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:38.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:38.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:38.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:38.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:38.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:38.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:38.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:38.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:38.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:38.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:38.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:38.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:38.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:38.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:38.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:38.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:38.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:39.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:39.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:39.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:39.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:39.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:39.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:39.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:39.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:39.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:39.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:39.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:39.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:39.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:39.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:39.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:40.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:40.184+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14213 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:40.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:40.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:40.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:40.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:40.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:40.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:40.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:40.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:40.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:40.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:40.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:40.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:40.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:40.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:40.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:40.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:40.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:40.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:40.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:40.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:41.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:41.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:41.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:41.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:41.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:41.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:41.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:42.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:41.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:42.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:41.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:42.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:41.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:42.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:41.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:42.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:42.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:42.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:42.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:42.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:42.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:42.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:42.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:42.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:42.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:42.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:42.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:42.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:42.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:42.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:42.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:42.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:42.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:42.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:42.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:43.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:43.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:43.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:43.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:43.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:43.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:43.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:43.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:43.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:43.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:43.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:43.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:43.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:43.659+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:43.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:43.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:43.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:43.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:43.982 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:43.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:44.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:44.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:44.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:44.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:44.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:44.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:44.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:44.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:44.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:44.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:44.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:44.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:44.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:44.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:44.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:44.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:44.611+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:45.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:45.186+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14283 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:45.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:45.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:46.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:45.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:46.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:45.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:46.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:45.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:46.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:45.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:46.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:45.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:46.024 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:45.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:46.024 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:45.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:46.024 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:45.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:46.024 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:45.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:46.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:46.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:46.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:46.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:46.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:46.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:46.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:46.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:46.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:46.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:46.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:46.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:46.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:46.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:46.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:46.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:46.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:46.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:46.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:46.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:47.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:47.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:47.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:47.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:47.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:47.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:47.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:47.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:47.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:47.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:47.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:47.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:47.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:47.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:47.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:47.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:47.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:47.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:47.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:47.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:48.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:48.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:48.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:48.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:48.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:48.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:48.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:48.588+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:48.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:48.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:48.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:48.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:48.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:48.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:49.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:48.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:49.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:48.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:49.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:48.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:49.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:49.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:49.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:49.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:49.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:49.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:49.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:49.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:49.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:49.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:49.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:49.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:49.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:49.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:50.180 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:49.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:50.180 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:49.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:50.180 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:49.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:50.442 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:50.188+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14355 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:50.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:50.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:50.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:50.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:50.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:50.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:50.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:50.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:50.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:50.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:50.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:50.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:50.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:50.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:50.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:51.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:51.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:51.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:51.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:51.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:51.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:51.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:51.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:51.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:51.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:51.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:51.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:51.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:51.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:51.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:51.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:52.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:52.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:52.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:52.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:52.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:52.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:52.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:52.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:52.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:52.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:52.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:52.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:52.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:52.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:52.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:52.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:52.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:52.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:52.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:52.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:53.750 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:53.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:53.751 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:53.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:53.751 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:53.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:53.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:53.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:53.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:53.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:53.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:53.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:53.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:53.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:53.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:53.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:53.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:53.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:53.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:53.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:54.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:54.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:54.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:54.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:54.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:54.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:54.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:54.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:54.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:54.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:54.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:54.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:54.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:54.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:54.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:54.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:54.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:54.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:54.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:54.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:55.463 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:30:55.191+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14425 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:30:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:55.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:55.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:55.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:55.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:55.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:55.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:55.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:55.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:55.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:55.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:55.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:55.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:55.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:56.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:56.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:56.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:56.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:56.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:56.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:56.817 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:56.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:56.817 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:56.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:56.817 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:56.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:56.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:56.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:56.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:56.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:56.818 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:56.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:56.818 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:56.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:57.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:57.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:57.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:57.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:57.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:57.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:57.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:57.528+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:57.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:57.528+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:57.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:57.528+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:57.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:57.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:57.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:57.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:57.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:57.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:57.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:57.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:58.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:58.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:58.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:58.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:58.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:58.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:58.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:58.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:58.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:58.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:58.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:58.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:58.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:58.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:58.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:58.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:58.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:59.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:59.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:59.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:59.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:59.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:30:59.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:30:59.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:59.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:59.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:59.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:59.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:30:59.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:30:59.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:59.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:59.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:59.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:59.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:59.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:30:59.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:30:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:30:59.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:00.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:00.193+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14497 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:00.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:00.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:00.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:00.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:00.737 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:00.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:01.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:00.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:01.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:00.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:01.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:00.734+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:01.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:00.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:01.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:00.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:01.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:00.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:01.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:00.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:01.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:01.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:01.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:01.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:01.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:01.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:02.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:01.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:02.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:01.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:02.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:01.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:02.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:01.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:02.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:01.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:02.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:01.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:02.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:01.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:02.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:02.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:02.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:02.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:02.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:02.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:02.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:02.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:02.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:02.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:02.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:02.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:02.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:02.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:02.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:02.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:02.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:02.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:02.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:02.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:03.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:03.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:03.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:03.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:03.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:03.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:03.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:03.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:03.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:03.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:03.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:03.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:03.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:03.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:04.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:04.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:04.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:04.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:04.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:04.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:04.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:04.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:04.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:04.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:04.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:04.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:04.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:04.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:04.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:04.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:04.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:05.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:05.194+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14567 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:05.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:05.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:05.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:05.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:06.038 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:05.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:06.039 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:05.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:06.039 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:05.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:06.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:05.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:06.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:05.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:06.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:05.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:06.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:05.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:06.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:06.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:06.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:06.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:06.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:06.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:07.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:06.802+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:07.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:06.802+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:07.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:06.802+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:07.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:06.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:07.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:06.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:07.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:06.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:07.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:06.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:07.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:07.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:07.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:07.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:07.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:07.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:08.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:07.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:08.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:07.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:08.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:07.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:08.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:07.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:08.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:07.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:08.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:07.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:08.231 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:07.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:08.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:08.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:08.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:08.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:08.700 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:08.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:08.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:08.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:08.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:08.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:08.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:08.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:08.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:08.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:08.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:08.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:08.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:08.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:08.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:08.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:09.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:09.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:09.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:09.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:09.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:09.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:09.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:09.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:09.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:09.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:09.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:09.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:09.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:09.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:09.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:09.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:09.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:10.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:10.196+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14639 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:10.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:10.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:10.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:10.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:10.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:10.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:10.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:10.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:10.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:10.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:10.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:10.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:10.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:10.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:10.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:10.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:11.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:11.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:11.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:11.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:11.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:11.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:11.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:11.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:11.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:11.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:11.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:11.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:11.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:11.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:11.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:11.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:11.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:11.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:11.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:11.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:12.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:12.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:12.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:12.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:12.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:12.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:12.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:12.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:12.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:12.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:12.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:12.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:12.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:12.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:12.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:12.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:12.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:12.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:12.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:13.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:13.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:13.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:13.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:13.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:13.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:13.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:13.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:13.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:13.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:13.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:13.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:13.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:13.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:13.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:13.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:13.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:13.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:13.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:13.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:14.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:14.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:14.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:14.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:14.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:14.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:14.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:14.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:14.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:14.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:14.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:14.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:14.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:15.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:15.199+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14709 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:15.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:15.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:15.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:15.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:15.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:15.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:15.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:15.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:15.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:15.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:15.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:15.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:15.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:15.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:15.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:15.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:15.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:15.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:15.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:15.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:16.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:16.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:16.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:16.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:16.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:16.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:16.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:16.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:16.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:16.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:16.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:16.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:16.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:16.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:16.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:16.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:16.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:16.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:16.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:16.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:17.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:17.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:17.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:17.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:17.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:17.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:17.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:17.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:17.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:17.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:17.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:17.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:17.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:17.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:17.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:17.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:17.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:17.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:17.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:17.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:18.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:18.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:18.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:18.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:18.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:18.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:18.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:18.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:18.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:18.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:18.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:18.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:18.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:18.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:18.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:18.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:19.673 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:19.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:19.673 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:19.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:19.673 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:19.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:19.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:19.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:19.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:19.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:19.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:19.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:19.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:20.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:20.201+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14781 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:20.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:20.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:20.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:20.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:20.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:20.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:20.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:20.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:20.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:20.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:20.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:20.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:20.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:20.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:20.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:20.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:21.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:21.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:21.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:21.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:21.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:21.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:21.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:21.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:21.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:21.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:21.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:21.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:21.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:21.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:21.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:21.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:21.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:22.648 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:22.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:22.648 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:22.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:22.648 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:22.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:22.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:22.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:22.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:22.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:22.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:22.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:22.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:22.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:22.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:22.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:22.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:23.620 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:23.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:23.621 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:23.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:23.621 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:23.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:23.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:23.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:23.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:23.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:23.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:23.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:23.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:23.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:23.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:23.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:23.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:23.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:23.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:23.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:24.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:24.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:24.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:24.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:24.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:24.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:24.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:24.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:24.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:24.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:25.208 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:24.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:25.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:24.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:25.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:24.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:25.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:25.204+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14851 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:25.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:25.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:25.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:25.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:25.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:25.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:25.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:25.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:25.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:25.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:25.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:25.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:25.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:26.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:26.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:26.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:26.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:26.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:26.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:26.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:26.770+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:26.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:26.770+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:26.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:26.770+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:26.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:26.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:26.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:26.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:27.647 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:27.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:27.647 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:27.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:27.647 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:27.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:27.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:27.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:27.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:27.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:27.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:27.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:27.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:27.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:27.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:27.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:27.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:28.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:28.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:28.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:28.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:28.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:28.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:28.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:28.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:28.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:28.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:28.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:28.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:28.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:28.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:28.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:28.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:28.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:29.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:29.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:29.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:29.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:29.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:29.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:29.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:29.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:29.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:29.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:29.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:29.779+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:29.779+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:29.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:29.779+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:30.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:30.207+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14923 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:30.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:30.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:30.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:30.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:30.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:30.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:30.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:30.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:30.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:30.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:30.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:30.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:30.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:30.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:30.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:30.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:30.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:30.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:30.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:31.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:31.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:31.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:31.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:31.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:31.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:31.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:31.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:31.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:31.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:31.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:31.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:31.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:32.688 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:32.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:32.689 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:32.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:32.689 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:32.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:32.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:32.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:32.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:32.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:32.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:32.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:32.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:32.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:32.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:32.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:33.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:33.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:33.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:33.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:33.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:33.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:33.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:33.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:33.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:33.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:33.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:33.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:33.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:33.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:33.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:33.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:33.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:33.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:34.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:34.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:34.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:34.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:34.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:34.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:34.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:34.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:34.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:34.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:34.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:34.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:34.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:34.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:34.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:35.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:35.209+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 14993 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:35.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:35.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:35.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:35.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:35.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:35.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:35.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:35.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:35.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:35.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:35.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:35.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:35.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:35.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:35.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:35.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:35.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:35.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:35.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:35.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:36.704 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:36.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:36.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:36.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:36.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:36.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:36.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:36.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:36.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:36.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:36.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:36.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:36.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:36.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:36.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:36.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:36.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:36.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:36.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:36.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:37.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:37.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:37.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:37.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:37.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:37.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:37.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:37.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:37.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:37.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:37.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:37.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:37.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:37.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:37.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:37.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:37.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:38.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:38.217+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:38.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:38.217+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:38.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:38.217+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:38.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:38.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:38.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:38.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:38.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:38.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:38.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:38.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:38.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:38.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:38.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:38.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:38.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:38.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:39.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:39.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:39.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:39.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:39.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:39.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:39.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:39.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:39.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:39.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:39.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:39.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:39.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:39.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:39.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:40.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:40.212+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15065 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:40.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:40.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:40.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:40.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:40.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:40.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:40.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:40.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:40.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:40.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:40.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:40.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:40.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:40.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:40.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:40.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:40.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:41.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:41.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:41.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:41.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:41.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:41.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:41.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:41.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:41.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:41.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:41.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:41.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:41.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:41.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:41.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:41.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:42.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:42.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:42.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:42.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:42.663 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:42.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:42.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:42.863+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:42.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:42.863+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:42.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:42.863+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:42.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:42.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:42.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:42.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:42.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:43.622 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:43.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:43.622 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:43.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:43.622 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:43.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:43.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:43.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:43.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:43.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:43.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:43.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:43.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:43.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:44.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:43.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:44.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:43.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:44.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:43.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:44.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:44.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:44.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:44.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:44.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:44.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:44.899 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:44.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:44.899 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:44.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:44.899 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:44.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:44.899 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:44.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:45.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:44.896+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:45.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:44.896+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:45.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:44.896+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:45.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:45.214+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15135 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:45.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:45.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:45.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:45.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:45.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:45.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:45.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:45.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:45.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:45.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:45.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:45.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:45.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:45.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:46.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:45.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:46.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:45.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:46.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:45.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:46.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:46.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:46.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:46.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:46.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:46.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:46.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:46.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:46.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:46.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:46.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:46.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:46.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:47.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:46.859+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:47.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:46.859+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:47.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:46.859+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:47.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:47.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:47.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:47.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:47.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:47.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:47.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:47.564+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:47.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:47.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:47.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:47.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:47.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:47.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:48.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:47.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:48.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:47.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:48.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:47.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:48.610 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:48.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:48.610 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:48.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:48.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:48.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:48.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:48.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:48.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:48.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:48.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:48.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:48.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:48.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:49.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:48.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:49.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:48.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:49.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:48.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:49.566 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:49.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:49.567 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:49.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:49.567 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:49.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:49.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:49.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:49.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:49.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:49.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:49.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:49.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:49.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:50.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:49.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:50.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:49.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:50.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:49.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:50.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:50.215+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15207 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:50.246+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:50.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:50.246+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:50.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:50.246+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:50.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:50.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:50.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:50.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:50.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:50.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:50.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:50.531+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:51.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:50.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:51.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:50.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:51.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:50.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:51.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:51.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:51.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:51.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:51.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:51.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:51.891 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:51.518+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:51.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:51.518+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:51.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:51.518+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:51.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:51.518+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:52.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:51.888+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:52.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:51.888+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:52.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:51.888+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:52.518 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:52.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:52.518 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:52.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:52.518 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:52.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:52.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:52.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:52.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:52.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:52.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:52.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:52.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:52.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:53.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:52.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:53.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:52.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:53.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:52.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:53.539 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:53.266+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:53.539 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:53.266+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:53.539 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:53.266+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:53.535+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:53.535+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:53.535+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:53.535+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:54.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:53.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:54.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:53.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:54.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:53.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:54.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:54.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:54.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:54.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:54.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:54.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:54.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:54.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:54.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:54.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:54.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:54.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:54.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:54.510+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:55.327 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:31:55.218+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15277 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:31:55.328 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:55.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:55.328 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:55.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:55.328 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:55.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:55.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:55.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:55.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:55.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:55.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:55.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:55.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:55.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:55.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:55.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:55.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:55.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:55.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:55.475+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:56.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:55.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:56.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:55.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:56.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:55.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:56.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:56.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:56.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:56.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:56.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:56.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:56.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:56.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:56.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:56.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:56.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:56.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:56.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:56.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:57.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:56.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:57.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:56.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:57.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:56.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:57.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:57.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:57.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:57.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:57.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:57.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:57.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:57.466+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:57.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:57.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:57.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:57.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:57.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:57.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:58.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:57.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:58.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:57.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:58.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:57.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:58.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:58.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:58.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:58.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:58.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:58.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:58.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:58.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:58.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:58.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:58.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:58.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:58.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:58.433+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:59.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:58.912+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:59.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:58.912+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:59.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:58.912+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:31:59.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:59.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:59.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:59.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:59.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:31:59.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:31:59.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:59.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:59.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:59.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:59.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:59.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:31:59.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:31:59.448+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:00.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:59.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:00.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:59.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:00.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:31:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:31:59.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:00.443 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:00.220+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15349 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:00.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:00.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:00.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:00.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:00.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:00.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:00.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:00.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:00.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:00.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:00.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:00.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:00.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:00.439+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:01.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:00.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:01.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:00.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:01.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:00.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:01.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:01.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:01.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:01.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:01.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:01.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:01.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:01.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:01.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:01.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:01.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:01.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:01.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:01.468+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:02.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:01.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:02.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:01.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:02.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:01.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:02.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:02.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:02.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:02.441+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:02.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:02.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:02.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:02.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:02.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:02.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:02.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:02.494+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:03.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:02.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:03.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:02.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:03.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:02.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:03.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:03.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:03.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:03.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:03.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:03.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:03.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:03.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:03.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:03.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:03.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:03.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:03.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:03.537+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:04.277 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:04.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:04.277 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:04.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:04.277 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:04.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:04.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:04.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:04.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:04.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:04.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:04.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:04.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:04.518+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:04.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:04.518+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:04.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:04.518+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:04.733 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:04.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:05.177 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:04.984+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:05.178 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:04.984+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:05.178 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:04.984+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:05.439 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:05.222+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15419 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:05.440 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:05.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:05.440 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:05.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:05.440 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:05.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:05.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:05.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:05.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:05.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:05.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:05.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:05.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:05.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:06.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:05.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:06.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:05.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:06.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:05.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:06.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:06.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:06.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:06.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:06.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:06.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:06.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:06.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:06.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:06.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:06.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:06.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:06.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:06.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:07.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:06.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:07.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:06.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:07.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:06.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:07.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:07.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:07.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:07.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:07.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:07.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:07.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:07.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:07.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:07.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:07.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:07.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:07.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:07.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:08.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:07.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:08.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:07.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:08.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:07.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:08.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:08.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:08.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:08.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:08.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:08.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:08.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:08.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:08.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:08.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:08.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:08.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:08.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:08.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:09.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:08.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:09.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:08.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:09.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:08.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:09.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:09.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:09.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:09.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:09.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:09.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:09.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:09.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:09.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:09.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:09.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:09.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:09.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:09.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:09.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:09.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:09.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:10.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:10.225+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15491 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:10.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:10.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:10.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:10.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:10.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:10.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:10.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:10.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:10.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:10.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:11.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:11.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:11.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:11.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:11.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:11.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:11.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:11.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:11.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:11.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:11.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:11.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:11.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:11.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:11.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:11.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:11.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:12.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:12.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:12.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:12.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:12.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:12.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:12.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:12.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:12.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:12.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:12.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:12.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:12.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:12.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:12.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:12.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:12.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:12.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:12.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:12.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:13.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:13.035+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:13.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:13.035+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:13.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:13.035+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:13.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:13.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:13.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:13.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:13.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:13.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:13.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:13.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:13.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:13.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:13.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:13.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:13.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:13.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:14.361 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:14.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:14.362 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:14.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:14.362 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:14.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:14.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:14.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:14.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:14.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:14.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:14.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:14.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:14.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:14.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:14.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:14.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:14.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:14.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:14.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:15.356 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:15.228+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15561 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:15.357 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:15.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:15.357 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:15.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:15.357 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:15.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:15.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:15.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:15.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:15.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:15.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:15.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:15.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:15.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:15.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:15.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:15.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:15.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:15.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:15.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:16.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:16.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:16.421 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:16.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:16.421 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:16.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:16.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:16.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:16.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:16.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:16.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:16.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:16.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:16.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:16.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:16.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:16.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:16.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:16.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:16.603+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:17.369 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:17.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:17.369 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:17.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:17.369 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:17.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:17.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:17.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:17.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:17.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:17.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:17.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:17.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:17.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:17.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:17.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:17.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:17.557+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:18.449 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:18.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:18.449 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:18.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:18.449 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:18.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:18.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:18.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:18.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:18.564+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:18.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:18.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:18.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:18.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:18.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:18.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:19.403 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:19.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:19.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:19.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:19.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:19.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:19.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:19.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:19.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:19.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:19.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:19.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:19.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:19.530+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:19.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:19.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:19.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:19.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:19.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:19.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:20.355 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:20.231+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15633 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:20.356 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:20.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:20.356 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:20.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:20.356 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:20.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:20.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:20.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:20.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:20.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:20.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:20.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:20.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:20.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:20.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:20.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:20.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:20.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:20.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:20.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:21.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:21.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:21.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:21.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:21.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:21.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:21.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:21.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:21.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:21.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:21.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:21.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:21.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:21.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:21.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:21.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:22.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:22.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:22.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:22.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:22.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:22.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:22.838 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:22.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:22.838 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:22.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:22.838 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:22.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:22.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:22.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:22.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:22.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:22.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:22.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:22.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:22.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:23.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:23.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:23.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:23.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:23.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:23.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:23.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:23.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:23.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:23.586+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:23.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:23.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:23.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:23.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:23.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:23.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:24.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:24.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:24.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:24.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:24.446 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:24.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:24.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:24.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:24.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:24.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:24.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:24.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:24.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:24.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:24.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:24.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:24.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:24.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:25.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:25.233+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15703 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:25.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:25.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:25.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:25.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:25.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:25.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:25.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:25.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:25.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:25.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:25.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:25.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:25.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:25.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:25.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:25.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:25.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:25.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:25.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:25.578+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:26.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:26.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:26.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:26.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:26.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:26.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:26.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:26.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:26.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:26.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:26.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:26.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:26.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:26.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:26.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:26.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:26.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:26.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:26.622+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:27.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:27.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:27.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:27.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:27.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:27.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:27.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:27.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:27.966 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:27.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:27.966 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:27.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:27.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:27.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:27.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:27.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:27.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:27.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:27.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:27.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:28.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:28.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:28.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:28.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:28.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:28.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:28.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:28.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:28.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:28.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:28.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:28.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:28.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:28.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:28.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:28.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:28.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:28.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:28.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:28.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:29.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:28.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:29.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:28.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:29.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:28.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:29.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:29.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:29.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:29.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:29.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:29.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:29.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:29.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:29.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:29.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:30.250 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:30.236+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15775 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:30.250 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:29.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:30.251 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:29.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:30.251 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:29.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:30.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:30.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:30.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:30.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:30.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:30.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:30.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:30.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:30.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:30.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:30.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:30.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:30.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:30.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:31.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:30.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:31.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:30.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:31.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:30.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:31.933 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:31.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:31.933 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:31.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:31.933 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:31.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:31.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:31.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:31.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:31.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:31.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:31.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:31.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:31.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:32.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:31.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:32.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:31.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:32.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:31.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:32.931 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:32.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:32.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:32.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:32.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:32.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:32.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:32.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:32.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:32.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:32.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:32.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:32.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:32.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:33.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:32.928+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:33.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:32.928+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:33.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:32.928+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:33.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:33.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:33.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:33.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:33.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:33.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:33.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:33.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:33.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:33.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:33.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:33.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:33.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:33.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:33.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:34.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:33.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:34.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:33.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:34.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:34.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:34.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:34.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:34.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:34.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:34.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:34.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:34.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:34.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:34.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:34.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:34.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:34.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:35.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:34.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:35.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:34.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:35.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:34.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:35.587 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:35.238+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15845 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:35.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:35.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:35.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:35.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:35.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:35.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:35.910 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:35.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:35.910 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:35.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:35.910 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:35.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:35.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:35.584+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:36.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:35.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:36.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:35.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:36.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:35.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:36.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:36.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:36.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:36.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:36.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:36.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:36.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:36.602+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:36.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:36.636+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:36.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:36.636+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:36.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:36.636+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:37.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:36.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:37.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:36.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:37.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:36.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:37.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:37.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:37.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:37.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:37.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:37.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:37.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:38.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:37.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:38.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:37.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:38.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:37.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:38.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:38.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:38.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:38.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:38.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:38.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:38.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:38.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:38.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:38.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:39.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:39.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:39.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:39.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:39.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:39.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:39.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:39.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:39.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:39.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:39.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:39.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:39.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:39.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:39.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:39.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:39.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:40.367 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:40.239+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15917 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:40.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:40.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:40.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:40.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:40.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:40.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:40.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:40.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:40.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:40.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:40.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:40.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:40.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:40.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:40.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:40.670+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:41.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:41.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:41.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:41.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:41.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:41.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:41.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:41.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:41.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:41.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:41.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:41.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:41.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:41.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:41.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:41.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:42.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:42.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:42.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:42.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:42.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:42.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:42.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:42.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:42.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:42.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:42.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:42.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:42.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:42.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:42.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:42.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:42.679+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:43.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:43.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:43.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:43.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:43.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:43.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:43.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:43.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:43.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:43.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:43.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:43.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:43.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:43.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:43.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:43.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:43.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:44.342 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:44.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:44.342 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:44.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:44.342 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:44.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:44.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:44.558+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:44.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:44.558+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:44.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:44.558+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:44.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:44.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:44.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:44.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:45.480 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:45.242+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 15987 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:45.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:45.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:45.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:45.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:45.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:45.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:45.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:45.564+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:45.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:45.564+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:45.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:45.564+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:45.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:45.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:45.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:45.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:45.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:45.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:45.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:45.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:46.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:46.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:46.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:46.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:46.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:46.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:46.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:46.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:46.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:46.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:46.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:46.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:46.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:46.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:46.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:46.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:46.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:46.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:46.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:46.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:47.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:47.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:47.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:47.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:47.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:47.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:47.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:47.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:47.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:47.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:47.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:47.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:47.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:47.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:47.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:47.581+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:48.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:48.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:48.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:48.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:48.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:48.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:48.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:48.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:48.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:48.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:48.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:48.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:48.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:48.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:49.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:49.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:49.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:49.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:49.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:49.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:49.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:49.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:49.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:49.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:49.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:49.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:49.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:49.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:49.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:50.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:50.245+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16059 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:50.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:50.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:50.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:50.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:50.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:50.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:50.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:50.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:50.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:50.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:50.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:50.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:50.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:51.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:51.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:51.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:51.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:51.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:51.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:51.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:51.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:51.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:51.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:51.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:51.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:51.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:51.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:51.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:51.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:51.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:51.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:51.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:51.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:52.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:52.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:52.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:52.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:52.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:52.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:52.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:52.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:52.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:52.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:52.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:52.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:52.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:52.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:52.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:52.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:52.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:52.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:52.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:52.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:53.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:53.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:53.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:53.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:53.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:53.683+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:53.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:53.683+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:53.986 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:53.683+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:53.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:53.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:53.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:53.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:53.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:53.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:53.987 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:53.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:54.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:54.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:54.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:54.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:54.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:54.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:54.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:54.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:54.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:54.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:54.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:54.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:54.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:54.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:54.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:54.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:54.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:54.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:54.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:55.376 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:32:55.247+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16129 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:32:55.376 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:55.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:55.376 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:55.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:55.376 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:55.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:55.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:55.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:55.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:55.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:55.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:55.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:55.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:55.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:55.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:55.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:55.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:55.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:56.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:56.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:56.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:56.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:56.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:56.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:56.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:56.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:56.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:56.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:56.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:56.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:56.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:56.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:56.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:56.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:56.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:56.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:56.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:57.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:57.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:57.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:57.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:57.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:57.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:57.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:57.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:57.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:57.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:57.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:57.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:57.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:57.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:57.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:57.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:57.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:57.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:58.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:58.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:58.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:58.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:58.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:58.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:58.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:58.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:58.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:58.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:58.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:58.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:58.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:58.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:58.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:58.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:58.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:58.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:58.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:58.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:59.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:59.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:59.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:59.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:59.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:32:59.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:32:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:59.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:59.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:59.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:32:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:32:59.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:59.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:59.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:59.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:59.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:32:59.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:32:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:32:59.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:00.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:00.250+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16201 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:00.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:00.262+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:00.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:00.262+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:00.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:00.262+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:00.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:00.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:00.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:00.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:00.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:00.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:00.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:00.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:00.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:00.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:00.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:00.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:00.722+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:01.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:01.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:01.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:01.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:01.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:01.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:01.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:01.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:01.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:01.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:01.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:01.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:01.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:01.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:01.709+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:01.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:01.709+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:01.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:01.709+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:32.591078+0000 (oldest deadline 2024-02-19T19:22:52.591078+0000) 2024-02-19T19:33:02.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:02.205+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:02.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:02.205+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:02.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:02.205+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:02.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:02.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:02.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:02.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:02.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:02.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:02.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:03.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:03.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:03.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:03.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:03.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:03.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:03.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:03.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:03.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:04.701 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:04.240+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:04.701 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:04.240+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:04.702 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:04.240+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:29.827660+0000 (oldest deadline 2024-02-19T19:22:49.827660+0000) 2024-02-19T19:33:04.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:04.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:04.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:04.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:04.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:04.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:04.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:04.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:05.665 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:05.253+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16271 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:05.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:05.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:05.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:05.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:06.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:06.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:06.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:06.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:07.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:07.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:07.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:07.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:08.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:08.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:08.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:08.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:08.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:08.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:09.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:09.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:09.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:09.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:09.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:09.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:10.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:10.256+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16343 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:10.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:10.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:10.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:10.611+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:11.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:11.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:11.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:11.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:11.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:11.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:11.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:12.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:12.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:12.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:12.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:12.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:12.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:12.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:13.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:13.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:13.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:13.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:14.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:14.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:14.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:14.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:14.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:14.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:22:43.267739+0000 (oldest deadline 2024-02-19T19:23:03.267739+0000) 2024-02-19T19:33:14.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:14.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:15.565 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:15.258+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16413 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:15.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:16.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:16.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:17.616+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:18.609+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:19.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:19.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:20.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:20.261+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16484 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:20.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:21.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:22.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:23.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:24.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:24.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:24.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:24.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:24.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:24.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:24.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:24.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:25.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:25.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:25.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:25.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:25.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:25.644 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:25.263+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16555 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:25.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:25.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:25.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:25.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:25.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:25.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:25.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:26.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:26.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:26.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:26.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:26.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:26.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:26.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:26.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:26.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:26.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:26.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:26.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:27.181+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:27.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:27.181+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:27.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:27.181+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:27.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:27.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:27.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:27.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:27.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:27.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:27.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:28.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:28.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:28.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:28.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:28.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:28.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:29.210 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:28.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:29.210 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:28.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:29.210 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:28.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:29.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:29.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:29.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:29.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:29.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:29.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:30.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:29.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:30.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:29.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:30.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:29.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:30.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:30.265+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16626 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:30.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:30.254+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:30.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:30.254+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:30.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:30.254+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:30.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:30.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:30.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:31.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:30.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:31.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:30.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:31.578 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:31.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:31.578 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:31.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:31.579 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:31.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:31.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:31.575+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:31.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:31.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:31.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:32.544 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:32.257+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:32.544 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:32.257+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:32.545 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:32.257+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:32.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:32.541+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:32.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:32.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:32.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:33.557 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:33.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:33.557 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:33.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:33.557 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:33.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:33.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:33.553+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:33.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:33.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:33.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:34.582 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:34.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:34.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:34.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:34.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:34.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:34.853 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:34.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:34.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:35.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:34.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:35.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:34.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:35.599 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:35.268+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16697 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:35.599 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:35.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:35.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:35.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:35.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:35.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:35.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:35.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:35.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:35.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:35.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:35.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:35.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:35.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:35.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:35.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:36.565 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:36.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:36.566 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:36.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:36.566 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:36.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:36.566 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:36.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:36.566 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:36.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:36.567 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:36.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:36.567 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:36.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:36.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:36.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:36.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:36.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:36.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:37.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:37.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:37.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:37.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:37.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:37.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:37.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:37.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:37.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:37.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:37.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:37.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:37.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:37.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:38.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:37.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:38.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:38.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:38.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:38.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:38.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:38.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:38.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:38.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:38.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:38.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:38.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:38.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:38.468+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:38.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:39.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:38.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:39.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:38.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:39.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:39.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:39.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:39.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:39.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:39.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:39.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:39.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:39.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:39.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:39.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:39.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:39.423+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:40.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:39.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:40.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:39.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:40.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:39.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:40.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:40.270+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16768 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:40.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:40.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:40.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:40.386+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:40.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:40.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:40.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:40.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:40.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:40.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:40.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:40.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:41.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:40.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:41.392+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:41.392+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:41.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:41.392+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:41.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:41.392+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:41.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:41.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:41.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:41.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:41.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:41.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:41.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:42.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:41.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:42.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:41.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:42.709 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:42.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:42.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:42.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:42.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:42.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:42.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:42.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:42.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:42.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:42.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:42.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:42.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:42.378+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:42.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:42.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:42.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:42.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:42.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:42.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:43.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:43.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:43.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:43.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:43.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:43.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:43.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:43.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:43.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:43.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:43.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:43.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:43.408+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:43.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:43.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:44.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:43.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:44.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:44.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:44.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:44.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:44.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:44.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:44.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:44.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:44.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:44.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:44.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:44.369+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:44.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:44.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:45.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:44.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:45.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:45.273+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16839 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:45.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:45.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:45.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:45.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:45.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:45.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:45.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:45.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:45.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:45.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:45.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:45.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:45.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:45.371+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:45.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:45.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:45.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:46.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:46.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:46.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:46.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:46.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:46.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:46.370+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:46.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:46.370+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:46.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:46.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:46.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:46.371+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:47.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:46.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:47.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:46.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:47.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:46.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:47.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:47.344+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:47.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:47.344+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:47.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:47.344+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:47.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:47.344+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:47.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:47.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:47.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:47.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:47.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:47.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:47.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:47.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:47.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:48.322+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:48.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:48.322+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:48.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:48.322+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:48.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:48.322+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:48.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:48.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:48.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:48.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:48.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:48.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:48.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:48.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:48.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:49.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:49.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:49.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:49.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:49.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:49.327+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:49.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:49.327+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:49.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:49.327+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:49.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:49.327+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:50.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:49.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:50.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:49.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:50.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:49.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:50.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:50.276+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16910 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:50.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:50.303+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:50.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:50.303+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:50.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:50.303+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:50.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:50.303+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:50.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:50.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:50.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:50.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:50.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:50.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:50.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:50.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:51.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:51.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:51.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:51.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:51.273+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:51.273+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:51.273+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:51.273+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:51.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:51.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:51.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:52.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:52.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:52.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:52.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:52.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:52.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:52.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:52.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:52.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:52.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:52.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:52.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:52.227+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:52.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:52.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:52.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:53.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:53.230+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:53.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:53.230+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:53.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:53.230+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:53.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:53.230+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:53.632 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:53.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:53.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:53.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:53.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:53.392+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:53.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:53.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:53.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:54.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:54.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:54.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:54.210+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:54.210+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:54.210+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:54.210+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:55.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:54.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:55.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:54.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:55.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:54.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:55.458 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:33:55.279+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 16982 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:33:55.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:55.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:55.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:55.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:55.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:55.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:55.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:55.192+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:55.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:55.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:55.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:55.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:55.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:55.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:56.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:55.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:56.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:55.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:56.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:55.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:56.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:56.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:56.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:56.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:56.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:56.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:56.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:56.182+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:56.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:56.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:56.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:56.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:56.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:56.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:57.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:56.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:57.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:56.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:57.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:56.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:57.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:57.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:57.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:57.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:57.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:57.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:57.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:57.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:57.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:57.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:57.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:58.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:57.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:58.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:57.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:58.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:57.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:58.120+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:58.120+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:58.120+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:58.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:58.120+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:58.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:58.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:58.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:58.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:58.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:58.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:58.697+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:58.697+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:58.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:58.697+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:59.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:59.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:59.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:59.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:59.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:59.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:33:59.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:33:59.090+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:59.678 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:59.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:59.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:59.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:59.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:33:59.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:33:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:59.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:59.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:33:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:33:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:33:59.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:00.405 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:00.281+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17054 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:00.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:00.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:00.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:00.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:00.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:00.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:00.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:00.078+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:00.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:00.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:00.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:00.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:00.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:00.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:00.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:00.650+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:00.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:00.650+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:00.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:00.650+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:01.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:01.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:01.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:01.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:01.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:01.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:01.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:01.079+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:01.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:01.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:01.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:01.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:01.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:01.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:01.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:01.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:01.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:01.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:02.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:02.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:02.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:02.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:02.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:02.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:02.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:02.114+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:02.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:02.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:02.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:02.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:02.589+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:02.589+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:02.589+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:03.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:03.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:03.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:03.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:03.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:03.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:03.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:03.093+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:03.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:03.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:03.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:03.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:03.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:03.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:03.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:03.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:03.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:03.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:03.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:03.571+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:04.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:04.108+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:04.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:04.108+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:04.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:04.108+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:04.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:04.108+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:04.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:04.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:04.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:04.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:04.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:04.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:04.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:04.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:04.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:04.731 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:04.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:05.413 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:05.284+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17126 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:05.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:05.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:05.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:05.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:05.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:05.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:05.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:05.154+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:05.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:05.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:05.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:05.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:05.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:05.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:05.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:05.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:05.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:05.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:06.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:06.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:06.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:06.188+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:06.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:06.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:06.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:06.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:06.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:06.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:06.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:06.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:06.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:06.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:06.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:06.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:07.543 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:07.228+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:07.228+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:07.228+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:07.229+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:07.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:07.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:07.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:07.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:07.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:07.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:07.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:08.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:08.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:08.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:08.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:08.183+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:08.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:08.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:08.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:08.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:08.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:08.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:08.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:08.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:08.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:08.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:08.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:09.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:09.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:09.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:09.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:09.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:09.134+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:09.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:09.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:09.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:09.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:09.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:09.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:09.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:10.413 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:10.287+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17198 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:10.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:10.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:10.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:10.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:10.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:10.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:10.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:10.134+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:10.666 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:10.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:10.667 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:10.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:10.667 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:10.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:10.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:10.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:10.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:11.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:11.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:11.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:11.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:11.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:11.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:11.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:11.154+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:11.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:11.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:11.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:11.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:11.732 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:11.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:11.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:11.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:11.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:11.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:11.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:11.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:12.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:12.151+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:12.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:12.151+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:12.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:12.151+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:12.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:12.151+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:12.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:12.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:12.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:12.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:12.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:12.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:12.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:12.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:12.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:13.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:13.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:13.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:13.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:13.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:13.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:13.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:13.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:13.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:13.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:13.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:13.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:13.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:13.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:13.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:13.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:13.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:14.091+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:14.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:14.091+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:14.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:14.091+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:14.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:14.091+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:14.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:14.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:14.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:14.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:14.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:14.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:14.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:14.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:14.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:15.368 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:15.288+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17270 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:15.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:15.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:15.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:15.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:15.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:15.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:15.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:15.086+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:15.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:15.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:15.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:15.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:15.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:15.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:15.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:15.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:15.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:16.064+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:16.064+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:16.064+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:16.065+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:16.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:16.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:16.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:16.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:16.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:16.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:16.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:16.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:16.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:16.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:16.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:16.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:17.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:17.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:17.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:17.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:17.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:17.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:17.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:17.080+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:17.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:17.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:17.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:17.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:17.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:17.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:17.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:17.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:17.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:18.127+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:18.127+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:18.127+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:18.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:18.127+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:18.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:18.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:18.544+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:18.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:18.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:18.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:18.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:18.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:18.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:19.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:19.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:19.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:19.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:19.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:19.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:19.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:19.171+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:19.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:19.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:19.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:19.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:19.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:19.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:20.408 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:20.291+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17342 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:20.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:20.136+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:20.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:20.136+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:20.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:20.136+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:20.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:20.136+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:20.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:20.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:20.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:20.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:20.708 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:20.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:20.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:20.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:20.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:21.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:21.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:21.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:21.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:21.089+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:21.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:21.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:21.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:21.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:21.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:21.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:22.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:22.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:22.051+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:22.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:22.051+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:22.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:22.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:22.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:22.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:22.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:22.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:22.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:22.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:23.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:23.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:23.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:23.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:23.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:23.049+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:23.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:23.050+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:23.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:23.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:23.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:23.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:23.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:23.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:23.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:23.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:23.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:24.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:24.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:24.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:24.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:24.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:24.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:24.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:24.018+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:24.779 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:24.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:24.779 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:24.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:24.779 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:24.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:25.037 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:24.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:25.037 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:24.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:25.038 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:24.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:25.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:25.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:25.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:25.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:25.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:25.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:25.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:25.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:25.717 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:25.294+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17414 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:25.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:25.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:25.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:25.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:25.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:25.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:25.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:25.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:25.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:26.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:26.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:26.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:26.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:26.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:26.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:26.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:26.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:26.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:26.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:27.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:27.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:27.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:27.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:26.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:27.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:27.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:27.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:27.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:27.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:27.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:27.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:27.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:27.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:27.945+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:28.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:28.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:28.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:28.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:28.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:29.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:28.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:28.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:28.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:28.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:28.922+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:29.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:29.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:29.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:30.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:29.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:30.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:29.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:30.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:29.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:30.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:29.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:30.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:29.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:30.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:29.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:30.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:29.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:30.448 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:30.297+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17486 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:30.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:30.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:30.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:30.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:30.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:30.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:30.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:30.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:30.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:30.910+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:31.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:30.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:31.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:30.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:31.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:30.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:31.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:31.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:31.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:31.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:31.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:31.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:31.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:31.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:31.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:31.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:31.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:31.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:31.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:32.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:32.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:32.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:32.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:32.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:32.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:33.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:32.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:33.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:32.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:33.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:32.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:33.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:32.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:33.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:33.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:33.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:33.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:33.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:33.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:33.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:33.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:33.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:34.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:33.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:34.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:33.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:34.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:33.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:34.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:33.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:34.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:34.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:34.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:34.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:34.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:34.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:34.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:34.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:34.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:34.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:34.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:34.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:34.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:35.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:35.299+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17558 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:35.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:35.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:35.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:35.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:35.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:36.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:36.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:36.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:36.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:35.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:35.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:35.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:35.950+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:36.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:36.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:36.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:36.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:36.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:36.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:37.041+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:37.041+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:37.041+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:37.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:36.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:37.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:36.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:37.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:36.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:37.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:36.909+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:37.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:37.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:37.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:38.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:38.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:38.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:37.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:37.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:37.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:37.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:38.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:38.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:38.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:39.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:39.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:39.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:39.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:38.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:39.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:38.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:39.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:38.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:39.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:38.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:39.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:39.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:39.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:40.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:39.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:40.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:39.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:40.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:39.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:40.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:39.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:40.433 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:40.303+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17630 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:40.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:40.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:40.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:40.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:40.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:40.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:40.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:40.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:40.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:41.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:40.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:41.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:40.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:41.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:40.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:41.079 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:40.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:41.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:41.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:41.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:41.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:41.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:41.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:41.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:41.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:41.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:42.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:41.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:42.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:41.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:42.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:41.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:42.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:41.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:42.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:42.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:42.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:42.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:42.429 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:42.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:42.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:42.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:42.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:43.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:42.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:43.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:42.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:43.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:42.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:43.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:42.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:43.404 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:43.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:43.404 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:43.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:43.404 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:43.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:43.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:43.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:43.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:44.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:43.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:44.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:43.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:44.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:43.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:44.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:43.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:44.418 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:44.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:44.418 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:44.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:44.418 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:44.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:44.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:44.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:44.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:45.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:44.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:45.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:44.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:45.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:44.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:45.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:44.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:45.433 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:45.306+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17702 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:45.433 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:45.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:45.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:45.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:45.434 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:45.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:45.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:45.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:45.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:45.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:45.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:45.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:46.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:45.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:46.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:45.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:46.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:45.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:46.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:45.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:46.378 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:46.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:46.378 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:46.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:46.378 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:46.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:46.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:46.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:46.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:47.146 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:46.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:47.146 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:46.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:47.146 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:46.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:47.146 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:46.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:47.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:47.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:47.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:47.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:47.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:47.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:47.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:47.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:47.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:47.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:47.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:47.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:48.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:47.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:48.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:47.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:48.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:47.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:48.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:47.899+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:48.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:48.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:48.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:48.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:48.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:48.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:48.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:48.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:48.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:48.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:48.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:48.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:48.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:49.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:49.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:49.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:49.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:49.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:49.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:49.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:49.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:49.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:50.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:49.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:50.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:49.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:50.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:49.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:50.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:49.866+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:50.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:50.309+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17774 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:50.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:50.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:50.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:50.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:50.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:50.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:50.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:50.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:50.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:50.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:51.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:51.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:51.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:51.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:51.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:51.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:51.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:51.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:51.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:51.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:51.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:51.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:51.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:52.287+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:52.287+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:52.287+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:52.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:52.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:52.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:52.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:52.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:52.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:52.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:52.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:52.809+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:53.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:53.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:53.636 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:53.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:53.636 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:53.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:53.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:53.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:53.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:53.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:53.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:53.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:53.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:53.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:53.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:53.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:54.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:54.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:54.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:54.219+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:54.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:54.219+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:54.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:54.219+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:54.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:54.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:54.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:54.813+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:55.698 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:34:55.311+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17846 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:34:55.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:55.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:55.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:55.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:55.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:55.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:55.700 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:55.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:55.700 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:55.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:55.700 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:55.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:55.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:55.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:55.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:55.808+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:56.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:56.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:56.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:56.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:56.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:56.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:56.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:56.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:56.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:56.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:56.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:56.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:56.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:56.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:56.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:57.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:56.825+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:57.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:57.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:57.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:57.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:57.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:57.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:57.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:57.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:57.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:57.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:57.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:57.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:57.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:58.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:58.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:58.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:58.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:58.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:58.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:58.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:58.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:58.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:59.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:58.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:59.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:58.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:59.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:58.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:34:59.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:58.837+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:59.321+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:59.321+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:34:59.321+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:34:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:59.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:59.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:59.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:34:59.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:34:59.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:00.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:59.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:00.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:59.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:00.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:59.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:00.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:34:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:34:59.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:00.440 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:00.313+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17918 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:00.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:00.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:00.442 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:00.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:00.442 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:00.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:00.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:00.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:00.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:00.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:00.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:00.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:01.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:00.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:01.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:00.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:01.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:00.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:01.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:00.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:01.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:01.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:01.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:01.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:01.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:01.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:01.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:01.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:01.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:02.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:02.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:02.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:02.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:02.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:02.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:02.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:01.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:02.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:01.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:02.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:01.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:02.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:01.879+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:02.581 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:02.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:02.581 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:02.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:02.581 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:02.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:03.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:03.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:03.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:02.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:02.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:03.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:02.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:03.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:02.860+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:03.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:03.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:03.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:03.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:03.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:03.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:04.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:04.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:04.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:03.897+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:04.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:03.897+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:04.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:03.897+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:04.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:03.897+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:04.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:04.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:04.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:05.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:05.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:05.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:04.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:04.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:04.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:04.905+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:05.653 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:05.316+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 17984 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:05.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:05.286+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:05.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:05.286+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:05.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:05.286+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:05.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:05.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:05.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:05.862+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:06.312 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:06.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:06.313 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:06.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:06.313 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:06.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:06.309+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:06.309+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:06.309+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:07.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:07.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:07.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:06.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:06.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:06.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:06.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:07.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:07.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:07.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:08.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:08.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:08.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:07.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:07.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:07.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:07.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:08.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:08.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:08.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:08.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:08.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:08.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:09.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:09.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:09.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:09.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:08.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:09.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:08.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:09.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:08.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:09.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:08.798+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:09.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:09.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:09.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:10.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:09.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:10.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:09.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:10.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:09.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:10.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:09.812+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:10.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:10.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:10.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:10.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:10.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:10.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:10.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:10.319+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18050 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:10.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:10.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:10.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:10.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:10.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:10.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:10.801+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:11.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:10.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:11.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:10.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:11.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:10.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:11.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:11.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:11.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:11.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:11.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:11.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:11.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:11.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:11.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:11.771+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:11.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:12.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:11.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:12.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:11.952+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:12.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:12.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:12.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:12.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:12.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:12.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:12.805+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:13.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:12.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:13.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:12.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:13.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:12.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:13.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:13.298+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:13.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:13.298+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:13.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:13.298+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:13.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:13.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:13.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:13.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:13.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:13.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:13.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:13.822+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:14.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:13.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:14.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:13.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:14.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:13.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:14.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:14.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:14.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:14.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:14.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:14.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:14.835+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:14.835+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:14.835+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:14.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:14.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:15.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:14.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:15.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:14.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:15.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:15.321+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18113 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:15.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:15.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:15.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:15.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:15.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:15.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:15.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:15.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:16.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:15.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:16.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:15.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:16.262+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:16.262+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:16.262+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:16.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:16.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:16.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:16.865+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:17.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:16.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:17.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:16.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:17.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:16.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:17.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:17.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:17.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:17.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:17.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:17.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:17.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:17.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:18.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:17.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:18.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:17.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:18.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:18.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:18.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:18.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:18.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:18.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:18.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:19.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:18.988+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:19.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:18.988+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:19.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:18.988+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:19.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:19.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:19.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:20.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:19.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:20.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:19.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:20.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:19.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:20.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:19.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:20.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:19.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:20.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:19.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:20.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:19.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:20.451 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:20.324+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18176 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:20.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:20.290+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:20.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:20.290+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:20.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:20.290+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:20.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:20.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:20.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:21.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:20.818+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:21.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:20.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:21.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:20.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:21.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:20.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:21.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:21.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:21.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:21.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:21.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:21.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:21.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:22.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:21.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:22.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:21.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:22.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:21.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:22.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:22.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:22.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:22.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:22.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:22.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:22.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:23.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:22.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:23.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:22.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:23.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:22.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:23.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:23.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:23.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:23.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:23.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:23.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:23.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:23.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:23.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:23.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:23.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:23.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:23.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:23.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:23.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:23.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:23.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:24.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:24.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:24.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:24.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:24.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:24.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:24.817+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:25.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:24.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:25.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:24.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:25.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:24.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:25.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:25.328+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18245 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:25.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:25.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:25.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:25.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:25.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:25.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:25.841+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:26.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:25.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:26.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:25.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:26.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:25.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:26.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:26.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:26.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:26.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:26.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:26.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:26.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:26.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:27.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:26.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:27.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:26.869+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:27.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:27.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:27.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:27.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:27.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:27.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:27.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:27.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:27.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:28.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:27.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:28.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:28.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:28.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:28.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:28.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:28.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:28.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:28.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:28.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:29.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:28.839+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:29.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:29.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:29.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:30.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:29.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:30.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:29.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:30.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:29.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:30.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:29.879+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:30.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:29.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:30.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:29.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:30.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:29.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:30.457 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:30.331+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18314 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:30.458 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:30.293+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:30.458 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:30.293+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:30.458 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:30.293+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:30.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:30.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:30.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:30.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:30.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:30.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:30.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:31.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:31.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:31.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:31.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:31.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:31.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:31.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:31.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:32.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:31.965+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:32.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:31.965+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:31.965+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:31.965+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:32.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:32.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:32.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:32.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:32.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:32.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:32.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:32.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:33.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:32.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:33.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:32.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:33.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:33.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:33.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:33.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:33.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:33.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:33.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:33.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:33.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:33.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:33.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:33.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:34.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:33.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:34.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:34.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:34.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:34.957+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:34.957+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:34.957+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:34.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:34.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:34.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:35.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:34.931+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:35.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:35.334+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18383 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:35.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:35.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:35.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:35.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:35.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:35.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:35.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:35.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:35.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:36.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:35.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:36.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:36.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:36.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:36.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:36.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:36.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:36.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:36.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:36.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:36.985+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:36.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:37.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:36.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:37.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:36.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:37.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:37.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:37.921 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:37.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:37.921 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:37.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:37.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:37.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:37.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:37.959+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:37.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:38.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:37.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:38.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:37.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:38.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:38.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:38.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:38.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:38.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:38.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:38.924+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:39.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:38.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:39.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:38.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:39.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:38.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:39.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:39.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:39.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:40.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:39.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:40.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:39.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:40.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:39.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:40.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:39.914+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:40.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:39.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:40.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:39.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:40.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:39.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:40.462 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:40.336+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18455 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:40.463 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:40.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:40.463 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:40.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:40.463 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:40.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:40.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:40.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:40.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:40.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:41.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:40.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:41.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:40.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:41.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:40.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:41.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:41.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:41.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:41.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:41.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:41.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:41.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:41.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:42.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:41.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:42.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:41.915+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:42.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:42.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:42.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:42.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:42.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:42.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:42.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:42.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:43.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:42.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:43.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:42.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:43.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:43.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:43.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:43.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:43.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:43.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:44.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:43.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:44.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:43.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:44.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:43.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:44.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:43.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:44.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:44.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:44.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:44.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:44.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:44.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:45.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:44.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:45.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:44.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:45.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:44.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:45.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:44.850+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:45.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:45.339+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18527 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:45.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:45.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:45.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:45.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:45.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:45.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:46.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:45.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:46.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:45.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:46.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:45.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:46.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:45.889+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:46.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:45.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:46.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:45.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:46.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:45.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:46.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:46.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:46.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:46.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:47.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:46.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:47.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:46.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:47.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:46.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:46.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:46.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:46.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:46.868+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:47.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:47.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:47.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:47.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:47.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:47.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:47.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:47.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:47.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:47.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:48.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:47.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:48.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:47.826+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:48.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:48.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:48.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:48.802+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:48.802+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:48.802+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:48.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:48.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:48.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:49.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:48.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:49.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:49.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:49.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:50.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:49.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:50.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:49.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:50.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:49.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:50.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:49.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:50.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:49.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:50.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:49.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:50.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:49.816+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:50.470 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:50.341+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18599 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:50.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:50.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:50.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:50.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:50.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:50.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:51.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:50.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:51.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:50.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:51.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:50.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:51.158 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:50.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:51.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:50.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:51.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:50.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:51.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:50.787+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:51.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:51.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:51.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:52.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:51.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:51.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:51.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:51.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:51.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:52.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:51.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:52.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:51.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:52.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:52.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:52.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:52.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:52.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:52.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:52.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:52.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:52.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:52.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:52.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:53.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:53.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:53.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:53.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:53.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:53.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:53.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:53.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:53.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:54.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:53.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:53.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:53.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:53.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:54.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:54.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:54.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:54.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:54.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:54.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:54.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:54.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:54.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:54.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:54.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:54.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:54.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:54.770+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:55.671 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:35:55.344+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18671 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:35:55.672 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:55.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:55.672 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:55.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:55.672 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:55.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:55.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:55.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:55.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:55.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:55.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:55.669+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:55.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:55.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:55.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:55.764+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:56.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:56.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:56.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:56.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:56.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:56.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:56.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:56.646+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:56.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:56.646+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:56.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:56.646+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:56.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:56.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:56.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:57.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:56.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:57.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:57.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:57.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:57.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:57.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:57.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:57.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:57.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:57.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:57.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:57.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:57.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:57.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:58.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:58.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:58.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:58.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:58.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:58.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:58.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:58.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:58.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:35:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:58.763+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:59.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:59.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:35:59.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:35:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:59.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:59.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:35:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:35:59.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:00.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:59.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:00.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:59.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:00.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:59.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:00.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:35:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:35:59.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:00.471 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:00.347+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18743 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:00.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:00.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:00.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:00.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:00.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:00.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:00.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:00.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:00.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:00.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:00.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:00.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:00.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:00.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:00.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:00.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:01.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:01.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:01.608 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:01.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:01.608 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:01.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:01.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:01.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:01.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:01.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:01.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:01.605+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:01.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:01.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:01.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:01.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:01.697+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:02.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:02.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:02.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:02.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:02.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:02.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:02.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:02.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:02.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:02.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:02.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:02.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:02.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:02.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:02.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:03.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:03.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:03.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:03.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:03.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:03.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:03.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:03.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:03.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:03.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:03.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:03.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:03.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:03.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:03.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:03.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:04.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:04.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:04.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:04.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:04.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:04.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:04.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:04.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:04.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:04.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:04.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:04.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:04.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:05.698 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:05.350+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18815 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:05.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:05.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:05.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:05.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:05.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:05.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:05.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:05.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:05.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:05.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:05.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:05.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:05.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:05.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:05.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:05.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:06.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:06.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:06.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:06.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:06.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:06.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:06.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:06.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:06.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:06.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:06.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:06.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:06.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:06.728+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:07.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:07.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:07.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:07.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:07.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:07.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:07.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:07.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:07.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:07.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:07.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:08.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:08.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:08.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:08.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:08.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:08.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:08.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:08.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:08.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:08.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:08.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:08.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:08.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:08.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:08.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:08.813+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:09.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:09.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:09.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:09.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:09.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:09.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:09.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:09.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:09.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:09.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:09.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:09.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:10.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:09.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:10.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:09.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:10.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:09.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:10.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:09.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:10.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:10.353+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18887 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:10.862 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:10.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:10.862 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:10.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:10.863 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:10.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:10.863 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:10.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:10.863 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:10.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:10.863 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:10.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:10.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:10.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:10.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:10.860+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:11.578 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:11.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:11.578 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:11.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:11.578 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:11.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:11.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:11.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:11.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:11.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:11.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:11.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:12.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:11.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:11.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:11.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:11.899+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:12.927 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:12.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:12.927 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:12.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:12.927 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:12.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:12.928 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:12.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:12.928 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:12.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:12.928 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:12.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:12.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:12.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:12.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:12.925+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:13.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:13.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:13.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:13.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:13.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:13.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:13.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:13.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:13.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:13.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:13.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:13.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:13.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:13.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:13.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:13.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:14.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:14.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:14.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:14.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:14.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:14.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:14.873 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:14.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:14.873 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:14.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:14.873 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:14.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:14.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:14.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:14.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:14.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:15.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:15.356+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 18959 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:15.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:15.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:15.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:15.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:15.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:15.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:15.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:15.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:15.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:15.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:15.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:15.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:15.822+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:16.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:16.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:16.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:16.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:16.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:16.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:16.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:16.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:16.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:16.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:16.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:16.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:16.782+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:17.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:17.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:17.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:17.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:17.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:17.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:17.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:17.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:17.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:17.791+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:18.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:18.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:18.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:18.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:18.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:18.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:18.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:18.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:18.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:18.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:19.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:19.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:19.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:19.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:19.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:19.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:19.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:19.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:19.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:20.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:19.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:20.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:19.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:20.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:19.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:20.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:19.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:20.475 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:20.359+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19031 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:20.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:20.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:20.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:20.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:20.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:20.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:20.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:20.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:20.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:20.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:21.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:21.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:21.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:21.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:21.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:21.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:21.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:21.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:21.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:21.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:21.789+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:22.540+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:22.540+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:22.540+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:22.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:22.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:22.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:22.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:23.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:22.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:22.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:22.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:22.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:23.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:23.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:23.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:23.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:23.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:23.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:23.757+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:23.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:23.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:23.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:23.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:23.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:23.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:24.782 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:24.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:24.783 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:24.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:24.783 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:24.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:24.783 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:24.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:24.783 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:24.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:24.783 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:24.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:24.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:24.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:24.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:24.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:25.685 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:25.362+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19102 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:25.686 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:25.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:25.686 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:25.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:25.686 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:25.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:25.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:25.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:25.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:25.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:25.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:25.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:25.759+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:26.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:26.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:26.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:26.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:26.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:26.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:26.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:26.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:26.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:26.728+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:27.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:27.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:27.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:27.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:27.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:27.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:27.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:27.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:27.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:27.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:28.785 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:28.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:28.785 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:28.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:28.785 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:28.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:28.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:28.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:28.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:28.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:28.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:28.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:28.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:28.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:28.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:28.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:29.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:29.447+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:29.447+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:29.447+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:29.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:29.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:29.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:30.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:29.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:30.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:29.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:30.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:29.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:30.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:29.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:30.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:30.365+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19175 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:30.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:30.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:30.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:30.765 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:30.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:30.765 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:30.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:30.765 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:30.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:30.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:30.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:30.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:30.763+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:31.668 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:31.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:31.669 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:31.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:31.669 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:31.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:31.669 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:31.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:31.669 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:31.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:31.669 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:31.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:31.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:31.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:31.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:31.782+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:32.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:32.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:32.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:32.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:32.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:32.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:32.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:32.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:32.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:32.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:32.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:32.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:32.812+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:33.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:33.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:33.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:33.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:33.641 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:33.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:33.641 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:33.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:33.641 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:33.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:33.641 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:33.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:33.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:33.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:33.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:33.860+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:34.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:34.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:34.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:34.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:34.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:34.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:34.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:34.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:34.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:34.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:34.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:34.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:34.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:35.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:35.367+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19246 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:35.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:35.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:35.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:35.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:35.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:35.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:35.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:35.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:35.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:35.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:35.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:35.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:35.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:35.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:36.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:35.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:36.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:35.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:36.502 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:36.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:36.502 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:36.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:36.502 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:36.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:36.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:36.499+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:36.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:36.499+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:36.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:36.499+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:36.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:36.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:36.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:36.811+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:37.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:37.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:37.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:37.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:37.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:37.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:37.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:37.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:37.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:37.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:37.850+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:38.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:38.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:38.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:38.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:38.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:38.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:38.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:38.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:38.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:38.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:39.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:39.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:39.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:39.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:39.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:39.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:39.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:39.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:39.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:40.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:39.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:40.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:39.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:40.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:39.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:40.206 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:39.830+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:40.461 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:40.370+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19319 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:40.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:40.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:40.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:40.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:40.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:40.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:40.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:40.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:40.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:41.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:40.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:40.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:40.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:40.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:41.576 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:41.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:41.577 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:41.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:41.577 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:41.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:41.577 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:41.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:41.577 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:41.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:41.577 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:41.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:41.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:41.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:41.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:41.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:42.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:42.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:42.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:42.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:42.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:42.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:42.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:42.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:42.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:42.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:42.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:42.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:42.920+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:43.641 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:43.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:43.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:43.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:43.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:43.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:43.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:43.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:43.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:43.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:43.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:43.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:44.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:43.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:44.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:43.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:44.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:43.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:44.233 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:43.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:44.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:44.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:44.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:44.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:44.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:44.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:45.374 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:44.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:45.374 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:44.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:45.374 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:44.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:45.375 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:44.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:45.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:45.373+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19390 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:45.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:45.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:45.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:45.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:45.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:45.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:45.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:45.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:45.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:45.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:45.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:45.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:45.952+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:46.618 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:46.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:46.619 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:46.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:46.619 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:46.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:46.619 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:46.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:46.619 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:46.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:46.619 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:46.383+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:46.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:46.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:46.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:46.925+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:47.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:47.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:47.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:47.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:47.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:47.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:47.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:48.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:47.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:47.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:47.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:47.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:48.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:48.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:48.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:48.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:48.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:48.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:48.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:48.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:48.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:48.845+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:49.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:49.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:49.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:49.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:49.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:49.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:50.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:49.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:50.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:49.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:50.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:49.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:50.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:49.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:50.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:50.376+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19463 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:50.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:50.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:50.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:50.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:50.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:50.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:51.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:50.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:51.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:50.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:50.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:50.839+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:51.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:51.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:51.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:51.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:51.661 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:51.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:51.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:51.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:51.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:51.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:51.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:51.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:51.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:52.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:52.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:52.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:52.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:52.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:52.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:53.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:52.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:53.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:52.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:53.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:52.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:53.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:52.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:53.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:53.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:53.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:53.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:53.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:53.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:53.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:53.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:53.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:53.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:53.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:53.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:54.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:54.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:54.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:54.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:54.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:54.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:54.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:54.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:54.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:54.892+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:54.892+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:55.512 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:36:55.379+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19534 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:36:55.512 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:55.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:55.512 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:55.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:55.512 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:55.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:55.513 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:55.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:55.513 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:55.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:55.513 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:55.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:55.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:55.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:55.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:55.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:56.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:56.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:56.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:56.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:56.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:56.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:56.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:56.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:56.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:56.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:56.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:56.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:56.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:56.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:56.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:56.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:57.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:57.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:57.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:57.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:57.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:57.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:57.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:57.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:57.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:57.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:57.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:57.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:57.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:58.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:58.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:58.307+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:58.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:58.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:58.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:59.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:58.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:59.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:58.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:58.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:36:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:58.850+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:59.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:59.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:36:59.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:36:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:59.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:59.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:36:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:36:59.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:00.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:59.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:00.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:59.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:00.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:59.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:00.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:36:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:36:59.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:00.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:00.382+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19607 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:00.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:00.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:00.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:00.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:00.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:00.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:00.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:00.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:00.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:00.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:00.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:00.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:00.833+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:01.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:01.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:01.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:01.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:01.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:01.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:01.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:01.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:01.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:01.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:01.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:01.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:01.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:02.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:02.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:02.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:02.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:02.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:02.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:02.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:02.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:02.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:02.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:02.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:03.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:03.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:03.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:03.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:03.643 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:03.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:03.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:03.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:03.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:03.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:03.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:03.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:03.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:03.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:03.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:03.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:04.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:04.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:04.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:04.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:04.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:04.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:04.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:04.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:04.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:04.831+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:05.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:05.386+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19678 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:05.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:05.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:05.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:05.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:05.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:05.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:05.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:05.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:05.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:05.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:05.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:05.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:06.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:06.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:06.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:06.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:06.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:06.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:06.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:06.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:06.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:06.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:07.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:07.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:07.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:07.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:07.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:07.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:07.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:07.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:07.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:07.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:08.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:08.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:08.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:08.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:08.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:08.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:08.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:08.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:08.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:08.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:08.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:08.753+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:09.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:09.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:09.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:09.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:09.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:09.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:09.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:09.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:09.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:09.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:09.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:09.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:09.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:09.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:09.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:09.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:10.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:10.389+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19751 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:10.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:10.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:10.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:10.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:10.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:10.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:10.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:10.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:10.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:10.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:11.547 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:11.231+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:11.547 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:11.231+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:11.548 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:11.231+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:11.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:11.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:11.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:11.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:11.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:11.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:11.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:11.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:11.829 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:11.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:11.829 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:11.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:11.829 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:11.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:12.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:12.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:12.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:12.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:12.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:12.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:12.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:12.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:12.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:12.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:12.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:12.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:12.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:13.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:13.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:13.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:13.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:13.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:13.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:13.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:13.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:13.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:13.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:14.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:14.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:14.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:14.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:14.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:14.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:14.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:14.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:14.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:14.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:14.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:15.393 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:15.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:15.393 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:15.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:15.394 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:15.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:15.668 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:15.392+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19822 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:15.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:15.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:15.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:15.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:15.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:15.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:15.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:15.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:15.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:15.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:15.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:15.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:15.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:15.666+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:16.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:16.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:16.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:16.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:16.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:16.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:16.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:16.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:16.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:16.560+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:16.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:16.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:16.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:16.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:16.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:17.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:17.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:17.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:17.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:17.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:17.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:17.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:17.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:17.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:17.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:17.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:18.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:18.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:18.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:18.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:18.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:18.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:18.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:18.550+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:18.550+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:18.550+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:19.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:19.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:19.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:19.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:19.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:19.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:19.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:19.736+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:19.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:19.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:19.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:20.422 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:20.395+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19895 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:20.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:20.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:20.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:20.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:20.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:20.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:20.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:20.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:20.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:21.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:20.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:21.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:20.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:21.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:20.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:21.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:20.730+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:21.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:21.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:21.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:21.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:21.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:21.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:21.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:21.910 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:21.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:21.910 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:21.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:21.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:21.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:21.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:21.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:21.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:21.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:22.170+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:22.170+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:22.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:22.170+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:22.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:22.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:22.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:22.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:22.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:22.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:22.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:23.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:23.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:23.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:23.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:23.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:23.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:23.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:23.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:23.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:23.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:23.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:23.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:23.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:23.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:23.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:23.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:24.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:24.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:24.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:24.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:24.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:24.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:24.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:24.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:24.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:24.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:24.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:24.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:25.399 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:25.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:25.400 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:25.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:25.400 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:25.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:25.694 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:25.397+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 19966 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:25.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:25.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:25.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:25.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:25.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:25.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:25.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:25.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:25.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:25.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:26.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:26.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:26.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:26.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:26.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:26.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:26.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:26.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:26.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:26.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:26.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:26.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:26.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:27.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:27.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:27.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:27.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:27.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:27.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:27.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:27.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:27.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:27.739 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:27.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:28.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:27.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:28.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:27.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:28.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:27.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:28.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:27.736+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:28.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:28.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:28.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:28.830 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:28.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:28.830 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:28.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:28.830 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:28.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:28.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:28.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:28.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:28.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:28.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:28.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:28.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:28.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:29.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:29.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:29.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:29.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:29.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:29.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:29.683+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:29.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:29.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:29.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:29.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:29.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:29.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:30.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:30.400+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20039 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:30.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:30.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:30.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:30.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:30.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:30.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:30.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:30.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:30.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:30.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:30.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:30.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:30.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:30.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:30.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:31.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:31.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:31.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:31.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:31.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:31.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:31.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:31.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:31.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:31.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:31.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:31.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:32.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:32.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:32.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:32.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:32.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:32.501+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:32.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:32.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:32.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:32.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:33.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:33.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:33.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:33.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:33.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:33.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:33.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:33.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:33.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:33.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:33.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:33.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:33.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:33.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:33.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:33.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:34.436 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:34.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:34.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:34.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:34.437 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:34.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:34.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:34.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:34.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:34.666+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:34.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:34.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:34.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:34.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:34.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:34.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:35.416 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:35.403+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20110 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:35.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:35.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:35.417 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:35.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:35.417 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:35.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:35.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:35.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:35.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:35.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:35.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:35.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:35.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:36.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:36.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:36.427 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:36.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:36.427 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:36.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:36.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:36.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:36.681 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:36.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:36.681 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:36.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:36.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:36.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:36.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:36.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:37.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:37.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:37.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:37.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:37.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:37.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:37.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:37.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:37.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:37.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:37.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:37.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:37.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:37.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:37.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:38.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:38.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:38.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:38.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:38.411 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:38.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:38.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:38.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:38.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:38.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:38.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:38.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:38.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:39.428 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:39.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:39.429 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:39.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:39.429 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:39.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:39.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:39.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:39.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:39.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:39.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:39.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:39.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:39.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:39.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:39.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:40.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:40.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:40.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:40.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:40.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:40.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:40.712 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:40.406+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20183 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:40.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:40.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:40.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:40.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:40.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:40.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:40.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:40.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:40.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:40.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:41.398 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:41.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:41.398 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:41.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:41.398 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:41.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:41.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:41.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:41.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:41.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:41.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:41.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:41.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:41.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:41.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:41.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:42.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:42.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:42.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:42.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:42.325 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:42.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:42.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:42.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:42.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:42.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:42.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:42.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:42.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:42.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:42.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:42.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:43.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:42.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:43.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:42.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:43.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:42.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:43.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:43.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:43.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:43.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:43.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:43.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:43.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:43.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:43.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:43.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:44.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:44.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:44.427 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:44.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:44.427 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:44.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:44.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:44.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:44.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:44.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:44.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:44.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:44.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:44.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:44.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:44.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:44.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:44.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:44.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:44.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:44.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:44.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:44.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:45.716 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:45.409+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20254 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:45.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:45.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:45.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:45.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:45.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:45.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:45.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:45.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:45.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:45.715+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:45.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:45.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:45.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:46.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:46.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:46.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:47.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:46.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:47.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:46.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:47.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:46.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:47.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:46.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:47.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:46.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:47.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:46.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:47.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:46.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:47.461+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:47.461+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:47.461+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:47.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:47.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:47.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:47.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:47.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:47.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:47.779+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:48.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:48.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:48.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:48.928+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:48.928+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:48.928+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:48.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:48.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:48.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:48.795+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:49.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:49.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:49.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:50.209 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:49.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:50.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:49.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:50.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:49.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:50.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:49.822+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:50.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:49.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:50.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:49.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:50.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:49.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:50.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:50.413+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20327 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:50.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:50.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:50.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:51.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:50.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:51.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:50.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:50.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:50.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:50.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:50.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:50.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:51.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:51.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:51.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:52.162 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:51.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:52.162 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:51.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:52.162 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:51.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:52.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:51.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:52.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:51.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:52.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:51.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:52.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:51.890+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:52.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:52.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:52.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:52.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:52.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:52.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:52.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:52.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:52.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:52.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:53.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:53.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:53.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:53.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:53.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:53.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:53.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:53.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:53.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:53.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:53.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:53.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:53.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:54.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:54.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:54.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:54.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:54.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:54.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:54.919+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:54.919+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:54.919+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:54.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:54.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:54.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:54.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:55.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:37:55.416+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20398 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:37:55.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:55.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:55.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:55.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:55.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:55.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:55.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:55.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:55.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:55.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:55.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:55.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:55.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:56.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:56.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:56.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:57.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:56.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:57.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:56.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:57.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:56.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:57.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:56.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:57.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:56.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:57.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:56.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:57.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:56.931+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:57.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:57.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:57.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:58.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:57.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:58.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:57.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:58.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:57.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:57.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:57.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:57.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:57.977+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:58.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:58.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:58.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:58.865+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:58.865+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:58.865+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:37:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:59.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:59.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:59.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:37:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:37:59.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:59.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:59.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:37:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:37:59.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:00.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:59.896+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:00.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:59.896+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:00.210 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:37:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:37:59.896+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:00.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:00.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:00.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:00.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:00.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:00.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:00.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:00.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:00.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:00.419+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20471 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:00.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:00.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:00.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:00.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:00.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:00.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:00.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:01.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:01.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:01.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:01.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:01.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:01.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:01.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:01.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:01.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:01.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:02.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:02.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:02.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:02.006+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:02.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:02.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:02.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:02.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:02.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:02.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:03.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:03.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:03.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:03.052+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:03.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:03.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:03.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:03.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:03.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:03.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:03.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.023+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:04.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:04.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:04.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:04.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:04.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:04.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:04.994+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:05.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:05.422+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20542 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:05.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:05.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:05.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:06.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:05.876+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:06.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:05.876+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:06.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:05.876+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.026+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:06.508 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:06.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:06.509 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:06.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:06.509 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:06.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:06.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:06.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:06.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:06.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:07.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:07.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:07.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:08.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:08.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:08.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:08.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:08.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:07.904+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:08.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:07.904+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:08.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:07.904+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:08.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:08.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:08.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:09.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:09.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:09.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:09.015+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:09.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:08.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:09.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:08.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:09.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:08.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:09.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:09.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:09.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:10.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:10.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:10.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:10.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:10.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:10.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:10.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:10.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:10.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:09.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:10.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:09.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:10.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:09.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:10.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:10.425+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20615 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:10.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:10.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:10.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:11.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:10.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:11.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:10.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:11.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:10.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:11.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:11.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:11.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:11.048+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:11.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:11.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:11.469+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:11.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:11.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:11.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:12.091+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:12.091+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:12.091+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:12.091+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:12.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:12.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:12.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:12.875+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:12.875+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:12.875+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:13.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:13.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:13.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:13.071+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:13.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:13.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:13.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:13.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:13.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:13.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:13.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:13.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:13.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:14.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:14.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:14.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:15.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:15.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:15.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:15.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:14.998+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:15.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:14.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:15.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:14.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:15.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:14.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:15.441 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:15.427+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20686 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:15.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:15.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:15.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:15.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:15.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:15.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:15.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:15.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:15.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:15.975+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:16.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:16.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:16.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:16.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:16.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:16.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:16.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:16.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:16.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:17.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:17.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:17.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:17.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:17.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:17.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:17.998+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:18.820 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:18.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:18.820 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:18.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:18.821 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:18.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:18.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:18.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:18.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:18.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:18.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:18.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:18.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:19.802 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:19.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:19.802 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:19.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:19.802 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:19.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:20.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:19.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:20.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:19.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:20.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:19.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:20.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:19.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:20.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:19.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:20.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:19.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:20.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:19.939+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:20.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:20.431+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20759 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:20.821 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:20.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:20.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:20.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:20.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:20.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:20.820+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:20.820+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:20.820+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:20.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:20.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:20.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:20.937+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:21.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:21.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:21.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:21.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:21.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:21.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:21.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:21.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:21.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:21.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:22.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:22.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:22.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:22.863+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:22.863+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:22.863+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:22.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:22.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:22.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:22.942+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:23.859 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:23.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:23.859 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:23.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:23.859 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:23.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:23.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:23.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:23.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:24.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:23.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:24.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:23.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:24.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:23.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:24.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:23.957+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:24.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:24.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:24.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:24.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:24.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:24.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:25.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:24.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:25.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:24.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:24.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:24.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:24.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:24.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:24.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:25.595 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:25.434+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20830 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:25.892 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:25.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:25.892 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:25.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:25.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:25.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:26.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:25.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:26.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:25.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:26.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:25.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:25.978+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:25.978+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:25.978+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:25.978+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:26.907 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:26.599+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:26.908 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:26.599+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:26.908 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:26.599+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:26.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:26.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:26.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:26.996+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:27.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:26.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:27.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:26.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:27.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:26.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:27.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:27.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:27.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:27.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:27.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:27.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:28.038+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:28.038+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:28.038+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:28.038+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:27.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:27.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:27.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:28.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:28.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:28.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:28.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:28.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:28.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.026+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:28.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:28.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:28.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:29.850 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:29.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:29.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:29.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:29.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:29.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:30.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:30.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:30.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:30.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:29.989+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:30.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:29.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:30.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:29.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:30.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:29.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:30.463 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:30.437+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20903 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:30.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:30.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:30.552+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:30.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:30.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:30.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:30.953+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:30.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:30.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:30.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:31.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:31.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:31.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:31.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:31.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:31.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:32.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:31.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:31.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:31.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:31.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:32.809 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:32.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:32.809 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:32.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:32.809 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:32.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:32.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:32.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:32.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:32.944+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:32.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:33.839 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:33.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:33.839 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:33.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:33.839 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:33.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:33.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:33.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:33.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:33.962+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:33.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:33.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:33.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:34.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:34.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:34.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:34.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:34.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:34.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:35.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:34.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:35.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:34.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:35.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:34.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:35.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:34.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:35.729 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:35.440+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 20974 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:35.639+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:35.639+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:35.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:35.639+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:35.766+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:35.766+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:35.766+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:36.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:35.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:36.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:35.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:36.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:35.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:36.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:35.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:36.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:36.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:36.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:36.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:36.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:36.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:36.966 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:36.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:36.966 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:36.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:36.966 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:36.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:36.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:36.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:36.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:36.963+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:37.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:37.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:37.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:37.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:37.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:37.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:37.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:37.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:37.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:37.981+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:38.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:38.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:38.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:38.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:38.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:38.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:39.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:39.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:39.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:39.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:39.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:39.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:39.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:39.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:39.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:39.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:39.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:39.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:39.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:39.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:39.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:40.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:40.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:40.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:40.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:40.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:40.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:40.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:40.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:40.710 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:40.444+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21047 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:40.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:40.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:40.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:40.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:40.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:40.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:40.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:41.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:41.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:41.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:41.041+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:41.713+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:41.713+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:41.713+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:41.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:41.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:41.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:41.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:42.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:42.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:42.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:42.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:42.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:42.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:42.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:42.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:42.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:42.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:42.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:42.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:42.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:43.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:43.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:43.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:43.076+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:43.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:43.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:43.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:43.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:43.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:43.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:44.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:44.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:44.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:44.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:45.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:44.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:45.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:44.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:45.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:44.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:45.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:44.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:45.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:44.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:45.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:44.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:45.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:45.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:45.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:45.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:45.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:45.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:45.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:45.046+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:45.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:45.448+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21118 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:45.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:45.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:45.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:45.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:45.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:45.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:46.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:46.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:46.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:46.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:46.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:46.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:46.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:46.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:46.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:46.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:46.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:46.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.020+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:47.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:47.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:47.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:47.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:47.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:47.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:47.987+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:48.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:48.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:48.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:48.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:48.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:48.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:48.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:48.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:48.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:49.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:48.943+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:50.213 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:49.872+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:50.213 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:49.872+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:50.213 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:49.872+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:50.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:49.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:50.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:49.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:50.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:49.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:50.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:49.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:50.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:49.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:50.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:49.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:50.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:49.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:50.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:50.451+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21191 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:50.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:50.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:50.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:50.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:50.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:50.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:50.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:50.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:50.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:51.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:50.914+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:51.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:51.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:51.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:51.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:51.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:51.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:51.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:51.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:51.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:52.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:51.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:52.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:52.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:52.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:52.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:52.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:52.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:52.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:52.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:52.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:52.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:52.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:52.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:52.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:52.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:52.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:52.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:53.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:53.866+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:53.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:53.866+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:53.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:53.866+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:53.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:53.866+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:53.905 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:53.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:53.905 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:53.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:53.905 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:53.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:53.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:53.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:53.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:54.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:55.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:54.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:55.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:54.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:54.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:54.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:54.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:54.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:54.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:54.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:54.914+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:55.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:38:55.454+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21262 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:38:56.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:55.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:56.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:55.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:56.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:55.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:56.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:55.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:56.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:55.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:56.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:55.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:56.053 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:55.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:56.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:55.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:56.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:55.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:56.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:55.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:56.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:56.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:56.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:57.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:56.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:57.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:56.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:57.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:56.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:57.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:56.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:57.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:56.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:57.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:56.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:57.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:56.917+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:57.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:57.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:57.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:57.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:57.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:57.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:57.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:57.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:57.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:57.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:58.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:58.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:58.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:38:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:58.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:58.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:58.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:38:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:58.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:58.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:58.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:38:59.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:38:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:38:58.997+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:00.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:00.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:00.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:00.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.032+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:00.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:59.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:00.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:59.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:00.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:38:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:38:59.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:00.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:59.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:00.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:59.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:00.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:38:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:38:59.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:00.469 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:00.457+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21335 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:00.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:00.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:00.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:00.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:02.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:01.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:02.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:01.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:02.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:01.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:02.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:01.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:02.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:01.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:02.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:01.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:02.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:02.043+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:02.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:02.043+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:02.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:02.043+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:02.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:02.043+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:03.071 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:02.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:03.071 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:02.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:03.071 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:02.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:03.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:02.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:03.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:02.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:03.072 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:02.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:03.069+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:03.069+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:03.069+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:03.069+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:03.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:03.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:03.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:03.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:03.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:03.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:03.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:03.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:03.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:04.082+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:04.082+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:04.082+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:04.083+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:04.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:04.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:04.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:04.940+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:04.940+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:04.940+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:05.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:05.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:05.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:05.112+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:05.587 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:05.460+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21406 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:05.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:05.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:05.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:06.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:06.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:06.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:06.124+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:06.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:05.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:06.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:05.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:06.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:05.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:07.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:06.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:07.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:06.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:07.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:06.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:07.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:06.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:07.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:06.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:07.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:06.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:07.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:07.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:07.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:07.113+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:08.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:07.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:08.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:07.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:08.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:07.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:08.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:07.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:08.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:07.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:08.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:07.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:08.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:08.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:08.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:08.112+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:09.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:08.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:09.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:08.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:09.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:08.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:09.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:08.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:09.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:08.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:09.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:08.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:09.150+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:09.150+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:09.150+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:09.150+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:10.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:09.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:10.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:09.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:10.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:09.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:10.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:09.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:10.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:09.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:10.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:09.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:10.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:10.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:10.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:10.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:10.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:10.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:10.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:10.146+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:10.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:10.463+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21479 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:11.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:10.931+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:11.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:10.931+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:11.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:10.931+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:11.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:10.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:11.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:10.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:11.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:10.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:11.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:11.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:11.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:11.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:11.173+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:12.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:11.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:12.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:11.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:12.183 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:11.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:12.183 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:11.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:12.183 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:11.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:12.183 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:11.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:12.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:12.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:12.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:12.181+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:13.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:12.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:13.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:12.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:13.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:12.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:13.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:12.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:13.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:12.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:13.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:12.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:13.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:13.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:13.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:13.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:13.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:13.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:13.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:13.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:13.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:13.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:13.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:13.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:13.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:14.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:14.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:14.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:14.113+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:15.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:14.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:15.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:14.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:15.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:14.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:15.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:14.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:15.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:14.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:15.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:14.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:15.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:15.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:15.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:15.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:15.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:15.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:15.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:15.065+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:15.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:15.466+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21550 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:15.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:15.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:15.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:16.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:16.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:16.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:16.067+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:15.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:15.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:15.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:17.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:16.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:17.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:16.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:17.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:16.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:17.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:16.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:17.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:16.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:17.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:16.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:17.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:17.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:17.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:17.093+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:17.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:17.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:17.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:17.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:17.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:17.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:18.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:18.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:18.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:18.077+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:19.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:19.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:19.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:18.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:18.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:18.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:19.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:19.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:19.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:19.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:20.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:19.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:20.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:19.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:20.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:19.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:20.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:20.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:20.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:20.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:20.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:20.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:20.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:20.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:20.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:20.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:20.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:20.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:20.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:20.033+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:20.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:20.469+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21623 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:20.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:20.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:20.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:21.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:21.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:21.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:22.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:22.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:22.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:21.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:21.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:21.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.977+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:21.977+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:23.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:23.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:23.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:23.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:22.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:22.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:22.957+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:22.957+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:22.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:22.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:22.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:23.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:23.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:23.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:23.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:23.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:23.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:24.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:24.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:24.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:24.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:23.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:23.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:23.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:24.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:23.954+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:25.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:24.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:25.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:24.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:25.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:24.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:25.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:24.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:25.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:24.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:25.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:24.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:25.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:24.917+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:25.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:25.172+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:25.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:25.172+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:25.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:25.172+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:25.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:25.472+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21694 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:26.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:25.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:26.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:25.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:26.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:25.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:26.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:25.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:26.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:25.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:26.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:25.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:26.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:25.940+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:26.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:26.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:26.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:26.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:26.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:26.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:26.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:26.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:26.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:26.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:27.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:27.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:27.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:27.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:27.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:27.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:27.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:27.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:27.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:27.981+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:28.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:28.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:28.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:28.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:28.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:28.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:28.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:28.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:28.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:28.975+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:29.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:29.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:29.301+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:30.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:29.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:30.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:29.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:30.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:29.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:30.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:29.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:30.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:29.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:30.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:29.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:30.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:29.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:30.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:30.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:30.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:30.749 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:30.477+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21767 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:30.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:30.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:30.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:30.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:30.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:30.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:30.987+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:31.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:31.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:31.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:32.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:31.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:32.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:31.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:32.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:31.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:32.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:32.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:32.308 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:32.308 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.002+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:32.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:32.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:32.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:32.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:32.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:32.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:32.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:32.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:32.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:32.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:32.998 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:32.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:32.999 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:32.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:33.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:33.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:33.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:33.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:33.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:33.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:33.939 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:33.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:33.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:33.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:33.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:33.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:33.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:33.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:33.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:33.939+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:34.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:34.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:34.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:34.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:34.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:34.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:34.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:34.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:34.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:34.943+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:34.943+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:34.943+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:34.943+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:35.607 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:35.480+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21838 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:35.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:35.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:35.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:35.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:35.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:35.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:35.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:35.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:35.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:36.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:35.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:36.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:35.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:36.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:35.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:36.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:35.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:36.696 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:36.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:36.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:36.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:36.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:36.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:36.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:36.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:36.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:36.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:36.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:36.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:36.976+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:36.976+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:36.976+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:36.976+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:37.694 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:37.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:37.694 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:37.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:37.694 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:37.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:37.693+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:37.693+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:37.693+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:38.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:38.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:38.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:38.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:38.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:38.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:38.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:38.017+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:38.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:38.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:38.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:38.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:38.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:38.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:38.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:38.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:38.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:39.332 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:39.053+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:39.333 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:39.053+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:39.333 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:39.053+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:39.333 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:39.054+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:39.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:39.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:39.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:39.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:39.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:39.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:39.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:39.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:39.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:39.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:40.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:40.102+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:40.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:40.102+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:40.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:40.102+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:40.380 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:40.102+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:40.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:40.483+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21910 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:40.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:40.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:40.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:40.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:40.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:40.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:41.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:41.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:41.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:41.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:41.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:41.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:41.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:41.060+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:41.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:41.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:41.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:41.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:41.674 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:41.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:41.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:41.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:41.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:42.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:42.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:42.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:42.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:42.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:42.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:42.379 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:42.093+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:42.702 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:42.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:42.702 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:42.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:42.702 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:42.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:42.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:42.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:42.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:43.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:43.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:43.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:43.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:43.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:43.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:43.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:43.135+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:43.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:43.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:43.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:43.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:43.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:43.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:43.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:43.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:43.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:44.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:44.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:44.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:44.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:44.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:44.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:44.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:44.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:44.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:44.119+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:44.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:44.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:44.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:44.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:44.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:45.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:45.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:45.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:45.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:45.152+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:45.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:45.152+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:45.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:45.486+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 21982 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:45.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:45.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:45.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:46.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:46.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:46.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:46.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:46.137+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:46.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:46.137+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:46.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:46.137+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:46.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:46.137+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:47.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:46.739+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:47.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:46.739+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:47.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:46.739+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:47.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:47.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:47.361+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:47.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:47.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:47.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:47.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:47.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:47.166+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:48.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:47.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:48.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:47.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:48.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:47.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:48.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:48.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:48.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:48.201+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:48.201+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:48.201+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:48.201+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:49.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:48.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:49.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:48.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:49.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:48.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:49.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:49.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:49.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:49.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:49.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:49.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:49.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:49.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:49.174+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:49.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:49.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:49.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:50.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:50.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:50.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:50.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:50.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:50.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:50.172+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:50.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:50.489+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22054 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:50.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:50.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:50.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:51.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:51.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:51.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:51.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:51.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:51.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:51.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:51.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:51.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:51.194+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:51.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:51.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:51.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:52.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:52.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:52.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:52.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:52.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:52.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:52.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:52.148+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:52.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:52.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:52.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:53.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:52.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:53.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:52.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:53.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:52.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:53.456 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:53.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:53.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:53.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:53.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:53.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:53.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:53.171+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:53.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:53.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:53.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:54.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:53.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:54.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:53.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:54.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:53.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:54.420 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:54.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:54.420 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:54.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:54.420 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:54.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:54.420 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:54.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:54.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:54.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:54.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:54.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:55.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:54.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:55.121 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:54.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:55.121 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:54.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:55.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:55.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:55.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:55.119+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:55.119+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:55.788 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:39:55.492+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22126 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:39:55.789 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:55.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:55.789 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:55.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:55.789 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:55.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:56.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:55.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:56.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:55.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:56.127 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:55.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:56.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:56.125+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:56.777 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:56.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:56.777 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:56.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:56.777 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:56.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:57.162 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:56.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:57.162 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:56.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:57.162 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:56.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:57.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:57.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:57.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:57.160+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:57.160+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:57.788 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:57.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:57.788 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:57.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:57.788 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:57.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:58.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:57.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:58.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:57.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:58.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:57.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:58.149+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:58.149+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:58.149+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:58.149+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:58.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:58.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:58.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:58.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:58.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:58.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:59.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:58.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:59.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:58.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:59.113 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:58.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:39:59.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:59.110+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:59.110+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:59.110+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:39:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:39:59.111+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:59.770 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:59.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:59.771 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:59.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:39:59.771 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:39:59.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:00.118 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:59.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:00.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:59.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:00.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:39:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:39:59.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:00.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:00.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:00.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:00.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:00.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:00.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:00.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:00.117+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:00.801 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:00.495+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22198 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:00.802 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:00.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:00.802 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:00.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:00.802 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:00.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:01.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:00.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:01.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:00.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:01.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:00.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:01.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:01.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:01.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:01.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:01.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:01.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:01.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:01.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:01.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:01.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:01.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:01.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:02.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:01.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:02.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:01.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:02.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:01.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:02.475 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:02.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:02.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:02.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:02.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:02.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:02.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:02.169+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:02.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:02.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:02.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:02.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:03.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:02.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:03.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:02.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:03.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:02.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:03.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:03.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:03.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:03.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:03.215+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:03.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:03.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:03.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:03.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:03.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:03.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:03.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:03.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:03.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:04.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:04.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:04.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:04.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:04.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:04.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:04.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:04.251+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:05.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:04.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:05.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:04.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:05.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:04.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:05.223+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:05.223+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:05.223+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:05.223+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:05.859 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:05.498+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22270 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:05.860 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:05.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:05.860 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:05.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:05.864 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:05.477+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:06.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:05.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:06.181 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:05.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:06.181 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:05.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:06.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:06.179+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:06.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:06.179+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:06.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:06.179+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:06.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:06.179+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:06.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:06.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:06.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:07.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:06.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:07.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:06.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:07.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:06.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:07.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:07.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:07.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:07.159+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:07.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:07.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:07.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:07.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:07.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:07.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:08.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:07.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:08.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:07.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:08.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:07.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:08.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:08.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:08.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:08.174+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:08.833 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:08.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:08.834 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:08.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:08.834 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:08.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:09.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:08.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:09.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:08.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:09.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:08.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:09.202+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:09.202+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:09.202+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:09.202+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:09.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:09.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:09.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:09.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:09.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:09.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:10.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:09.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:10.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:09.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:10.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:09.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:10.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:10.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:10.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:10.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:10.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:10.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:10.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:10.194+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:10.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:10.502+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22342 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:10.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:10.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:10.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:11.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:10.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:11.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:10.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:11.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:10.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:11.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:11.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:11.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:11.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:11.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:11.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:11.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:11.172+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:11.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:11.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:11.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:11.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:12.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:11.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:12.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:11.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:12.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:11.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:12.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:12.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:12.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:12.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:12.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:12.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:12.220+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:12.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:12.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:12.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:13.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:13.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:13.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:13.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:13.656 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:13.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:13.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:13.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:13.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:13.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:13.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:13.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:13.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:13.250+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:13.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:13.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:13.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:14.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:14.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:14.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:14.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:14.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:14.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:14.261+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:15.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:14.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:14.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:14.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:15.635 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:15.505+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22414 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:15.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:15.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:15.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:15.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:15.636 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:15.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:15.636 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:15.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:15.637 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:15.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:15.637 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:15.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:15.637 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:15.286+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:15.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:15.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:15.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:15.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:15.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:15.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:16.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:16.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:16.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:16.305+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:16.305+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:16.305+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:16.306+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:17.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:16.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:16.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:16.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:17.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:17.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:17.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:17.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:17.349+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:17.349+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:17.349+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:17.349+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:18.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:17.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:17.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:17.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:18.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:18.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:18.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:18.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:18.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:18.323+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:18.323+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:18.323+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:18.323+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:19.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:18.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:18.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:18.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:19.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:19.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:19.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:19.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:19.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:19.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:19.372+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:20.219 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:19.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:20.219 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:19.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:20.219 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:19.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:20.413+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:20.413+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:20.413+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:20.413+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:20.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:20.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:20.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:20.806 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:20.509+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22486 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:21.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:20.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:20.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:20.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:21.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:21.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:21.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:21.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:21.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:21.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:21.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:21.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:21.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:22.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:21.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:22.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:21.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:22.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:21.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:22.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:22.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:22.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:22.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:22.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:22.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:22.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:22.505+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:23.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:22.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:22.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:22.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:23.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:23.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:23.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:23.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:23.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:23.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:23.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:23.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:23.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:23.506+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:23.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:23.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:23.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:24.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:24.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:24.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:24.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:24.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:24.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:24.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:25.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:24.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:25.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:24.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:25.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:24.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:25.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:25.513+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22558 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:25.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:25.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:25.399+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:25.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:25.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:25.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:25.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:26.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:25.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:25.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:25.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:26.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:26.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:26.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:26.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:26.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:26.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:26.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:27.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:26.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:26.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:26.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:27.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:27.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:27.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:27.402+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:27.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:27.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:27.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:27.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:27.569+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:27.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:27.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:27.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:28.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:28.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:28.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:28.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:28.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:28.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:28.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:28.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:28.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:28.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:29.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:29.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:29.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:29.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:29.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:29.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:29.511+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:30.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:29.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:30.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:29.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:30.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:29.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:30.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:30.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:30.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:30.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:30.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:30.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:30.516+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22630 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:30.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:30.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:30.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:30.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:30.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:30.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:30.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:30.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:30.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:30.513+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:31.563 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:31.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:31.563 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:31.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:31.564 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:31.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:31.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:31.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:31.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:31.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:31.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:31.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:31.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:31.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:31.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:32.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:32.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:32.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:32.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:32.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:32.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:32.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:32.588+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:33.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:32.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:32.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:32.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:33.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:33.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:33.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:33.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:33.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:33.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:33.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:33.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:33.623+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:34.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:33.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:33.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:33.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:34.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:34.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:34.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:34.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:34.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:34.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:34.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:34.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:34.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:34.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:34.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:35.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:35.519+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22702 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:35.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:35.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:35.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:35.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:35.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:35.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:35.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:35.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:35.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:35.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:36.637 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:36.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:36.638 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:36.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:36.638 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:36.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:36.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:36.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:36.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:36.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:36.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:36.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:36.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:37.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:37.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:37.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:37.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:37.674 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:37.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:37.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:37.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:37.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:37.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:37.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:37.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:37.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:38.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:38.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:38.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:38.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:38.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:38.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:38.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:38.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:38.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:38.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:38.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:38.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:38.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:39.675 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:39.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:39.675 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:39.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:39.675 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:39.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:39.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:39.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:39.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:39.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:39.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:39.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:39.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:39.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:40.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:40.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:40.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:40.884 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:40.522+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22774 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:40.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:40.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:40.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:40.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:40.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:40.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:40.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:40.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:40.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:40.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:40.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:41.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:41.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:41.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:41.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:41.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:41.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:41.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:41.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:41.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:41.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:41.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:41.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:41.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:41.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:41.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:41.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:41.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:42.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:42.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:42.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:42.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:42.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:42.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:42.611+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:43.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:42.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:42.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:42.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:43.604 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:43.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:43.604 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:43.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:43.605 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:43.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:43.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:43.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:43.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:43.602+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:44.381 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:43.986+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:44.381 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:43.986+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:44.381 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:43.986+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:44.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:44.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:44.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:44.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:44.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:44.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:44.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:44.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:45.397 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:45.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:45.398 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:45.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:45.398 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:45.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:45.650 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:45.525+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22846 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:45.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:45.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:45.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:45.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:45.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:45.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:45.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:45.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:45.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:45.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:45.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:45.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:45.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:45.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:46.359 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:45.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:46.360 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:45.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:46.360 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:45.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:46.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:46.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:46.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:46.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:46.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:46.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:46.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:46.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:46.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:46.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:46.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:46.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:46.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:46.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:47.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:46.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:46.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:46.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:47.669 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:47.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:47.670 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:47.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:47.670 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:47.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:47.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:47.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:47.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:47.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:47.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:47.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:47.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:48.664 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:48.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:48.665 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:48.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:48.665 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:48.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:48.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:48.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:48.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:48.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:48.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:48.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:48.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:48.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:49.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:48.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:48.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:48.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:49.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:49.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:49.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:49.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:49.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:49.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:49.650+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:50.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:49.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:50.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:49.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:50.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:49.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:50.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:50.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:50.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:50.967 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:50.528+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22918 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:50.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:50.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:50.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:50.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:50.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:50.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:51.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:50.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:50.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:50.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:51.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:51.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:51.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:51.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:51.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:51.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:51.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:51.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:51.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:51.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:52.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:51.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:52.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:51.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:52.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:51.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:52.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:52.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:52.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:52.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:52.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:52.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:52.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:52.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:52.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:52.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:52.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:53.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:53.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:53.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:53.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:53.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:53.008+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:53.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:53.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:53.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:53.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:53.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:53.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:53.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:54.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:53.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:53.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:53.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:54.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:54.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:54.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:54.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:54.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:54.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:54.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:54.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:55.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:54.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:55.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:54.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:54.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:55.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:40:55.531+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 22990 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:40:55.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:55.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:55.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:55.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:55.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:55.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:55.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:55.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:55.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:55.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:55.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:56.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:55.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:55.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:55.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:56.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:56.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:56.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:56.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:56.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:56.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:56.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:56.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:56.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:56.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:57.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:57.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:57.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:57.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:57.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:57.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:57.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:57.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:57.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:57.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:57.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:58.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:58.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:58.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:58.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:58.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:58.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:58.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:58.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:58.996+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:58.996+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:58.996+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:40:59.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:59.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:59.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:59.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:59.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:40:59.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:40:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:59.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:59.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:59.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:40:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:40:59.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:00.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:59.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:59.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:40:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:40:59.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:00.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:00.535+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23062 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:00.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:00.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:00.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:00.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:00.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:00.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:00.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:00.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:00.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:00.584+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:01.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:01.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:01.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:01.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:01.429 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:01.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:01.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:01.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:01.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:01.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:01.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:01.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:01.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:01.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:02.404 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:01.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:02.404 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:01.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:02.404 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:01.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:02.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:02.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:02.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:02.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:02.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:02.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:02.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:02.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:03.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:02.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:03.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:02.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:03.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:02.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:03.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:03.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:03.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:03.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:03.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:03.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:03.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:03.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:03.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:03.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:03.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:04.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:04.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:04.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:04.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:04.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:04.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:04.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:04.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:04.900+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:04.900+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:04.900+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:05.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:05.539+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23134 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:05.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:05.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:05.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:05.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:05.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:05.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:05.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:06.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:05.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:05.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:05.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:06.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:06.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:06.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:06.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:06.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:07.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:06.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:07.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:06.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:06.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:07.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:07.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:07.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:07.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:07.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:07.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:07.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:07.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:07.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:07.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:07.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:07.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:07.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:07.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:08.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:07.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:07.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:07.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:08.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:08.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:08.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:08.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:08.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:08.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:08.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:08.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:08.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:08.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:08.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:08.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:08.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:08.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:08.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:08.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:08.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:09.922 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:09.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:09.922 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:09.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:09.924 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:09.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:09.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:09.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:09.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:09.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:09.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:09.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:09.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:09.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:10.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:09.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:10.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:09.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:10.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:09.921+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:10.925 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:10.543+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23206 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:10.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:10.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:10.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:10.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:10.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:10.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:10.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:10.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:10.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:10.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:10.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:10.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:10.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:10.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:11.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:10.924+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:11.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:10.924+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:11.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:10.924+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:11.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:11.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:11.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:11.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:11.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:11.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:11.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:11.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:11.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:11.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:11.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:11.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:11.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:11.679+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:12.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:11.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:12.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:11.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:12.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:11.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:12.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:12.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:12.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:12.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:12.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:12.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:12.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:12.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:12.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:12.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:12.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:12.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:12.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:12.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:13.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:12.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:12.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:12.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:13.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:13.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:13.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:13.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:13.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:13.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:13.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:13.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:13.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:13.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:13.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:13.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:13.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:13.728+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:14.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:13.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:13.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:13.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:14.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:14.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:14.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:15.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:14.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:15.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:14.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:14.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:14.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:14.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:14.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:14.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:15.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:15.547+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23278 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:15.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:15.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:15.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:16.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:15.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:15.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:15.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:15.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:15.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:15.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:15.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:16.744 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:16.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:16.744 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:16.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:16.744 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:16.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:17.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:16.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:16.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:16.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:16.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:16.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:16.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:17.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:16.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:17.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:17.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:17.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:17.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:17.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:17.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:17.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:17.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:17.868 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:17.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:17.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:17.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:17.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:17.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:18.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:17.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:17.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:17.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:18.790 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:18.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:18.790 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:18.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:18.790 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:18.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:19.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:18.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:18.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:18.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:18.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:18.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:18.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:18.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:19.771 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:19.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:19.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:19.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:19.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:19.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:20.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:19.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:20.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:19.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:20.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:19.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:20.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:19.771+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:20.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:19.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:20.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:19.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:20.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:19.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:20.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:20.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:20.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:20.818 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:20.551+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23350 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:21.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:20.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:20.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:20.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:20.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:20.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:20.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:20.817+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:21.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:21.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:21.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:21.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:21.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:21.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:21.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:21.871+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:21.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:21.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:21.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:21.823+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:22.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:22.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:22.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:23.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:22.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:22.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:22.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:23.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:22.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:23.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:22.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:22.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:22.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:23.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:23.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:23.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:23.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:23.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:23.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:24.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:23.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:23.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:23.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:23.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:23.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:23.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:24.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:23.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:24.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:24.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:24.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:24.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:25.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:24.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:24.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:24.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:25.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:24.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:25.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:24.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:24.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:24.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:25.889 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:25.554+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23422 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:25.889 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:25.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:25.889 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:25.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:25.889 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:25.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:25.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:25.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:25.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:26.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:25.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:25.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:25.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:26.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:25.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:26.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:26.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:26.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:26.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:27.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:26.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:27.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:26.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:26.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:26.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:26.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:26.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:26.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:27.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:27.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:27.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:27.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:27.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:27.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:28.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:28.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:28.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:28.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:27.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:27.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:27.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:27.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:28.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:28.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:28.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:28.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:28.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:28.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:29.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:28.996+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:29.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:28.996+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:28.996+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:28.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:28.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:28.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:28.795+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:29.818 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:29.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:29.818 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:29.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:29.818 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:29.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:30.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:29.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:30.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:29.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:30.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:29.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:30.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:29.817+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:30.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:30.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:30.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:30.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:30.557+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23494 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:30.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:30.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:30.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:30.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:30.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:30.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:30.799+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:31.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:31.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:31.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:31.000+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:31.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:31.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:31.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:31.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:31.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:31.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:31.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:31.759+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:32.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:32.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:32.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:32.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:32.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:32.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:32.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:32.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:32.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:32.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:32.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:32.772+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:33.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:32.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:32.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:32.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:33.945 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:33.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:33.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:33.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:33.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:33.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:33.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:33.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:33.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:33.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:33.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:33.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:33.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:33.733+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:34.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:33.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:33.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:33.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:34.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:34.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:34.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:34.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:34.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:34.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:34.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:34.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:34.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:34.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:34.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:34.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:34.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:34.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:34.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:34.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:34.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:35.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:35.561+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23566 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:35.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:35.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:35.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:35.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:35.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:35.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:35.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:36.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:36.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:36.016+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:36.016+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:36.774 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:36.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:36.774 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:36.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:36.774 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:36.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:37.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:37.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:37.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:37.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:36.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:36.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:36.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:36.773+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:37.806 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:37.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:37.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:37.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:37.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:37.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:38.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:38.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:38.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:38.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:37.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:37.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:37.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:37.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:38.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:38.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:38.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:38.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:38.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:38.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:38.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:38.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:39.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:39.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:39.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:39.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:39.829 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:39.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:39.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:39.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:39.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:39.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:40.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:40.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:40.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:40.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:39.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:40.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:39.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:40.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:39.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:40.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:39.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:40.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:40.564+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23638 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:40.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:40.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:40.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:40.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:40.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:40.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:40.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:41.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:41.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:41.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:41.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:41.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:41.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:41.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:41.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:41.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:41.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:41.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:41.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:41.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:41.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:41.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:41.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:41.788+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:42.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:42.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:42.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:42.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:42.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:42.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:42.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:42.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:42.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:42.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:42.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:42.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:42.759+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:43.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:42.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:42.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:42.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:43.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:43.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:43.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:43.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:43.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:43.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:43.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:43.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:43.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:43.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:43.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:43.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:43.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:43.761+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:44.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:43.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:44.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:43.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:43.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:44.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:44.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:44.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:44.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:44.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:44.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:44.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:44.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:44.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:44.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:44.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:44.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:44.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:44.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:45.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:44.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:44.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:44.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:45.953 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:45.568+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23711 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:45.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:45.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:45.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:45.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:45.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:45.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:45.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:45.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:45.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:45.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:45.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:45.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:45.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:45.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:46.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:45.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:45.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:45.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:46.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:46.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:46.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:46.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:46.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:46.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:46.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:46.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:47.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:46.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:46.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:46.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:47.962 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:47.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:47.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:47.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:47.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:47.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:47.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:47.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:47.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:47.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:47.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:47.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:47.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:47.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:48.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:47.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:48.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:47.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:48.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:47.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:48.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:48.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:48.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:48.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:48.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:48.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:48.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:48.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:49.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:49.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:49.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:49.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:49.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:49.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:49.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:49.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:49.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:49.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:49.753+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:50.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:50.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:50.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:50.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:50.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:50.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:50.571+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23784 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:50.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:50.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:50.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:50.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:50.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:50.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:50.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:51.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:51.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:51.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:51.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:51.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:51.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:51.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:51.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:51.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:51.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:51.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:52.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:52.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:52.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:52.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:52.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:52.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:52.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:52.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:52.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:52.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:52.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:52.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:52.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:52.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:52.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:53.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:53.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:53.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:53.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:53.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:53.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:53.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:53.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:53.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:53.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:53.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:53.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:53.881+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:54.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:54.013+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:54.013+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:54.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:54.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:54.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:54.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:54.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:54.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:54.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:54.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:54.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:54.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:55.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:55.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:55.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:55.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:55.846 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:41:55.576+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23856 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:41:55.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:55.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:55.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:55.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:55.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:55.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:56.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:56.085+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:56.085+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:56.085+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:55.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:55.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:55.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:55.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:56.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:56.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:56.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:56.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:56.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:56.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:56.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:56.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:56.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:56.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:56.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:57.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:57.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:57.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:57.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:57.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:58.117 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:57.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:58.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:57.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:58.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:57.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:58.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:57.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:58.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:57.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:58.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:57.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:58.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:57.811+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:58.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:58.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:58.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:58.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:58.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:58.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:58.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:58.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:58.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:58.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:41:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:58.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:58.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:58.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:41:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:58.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:59.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:59.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:41:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:41:59.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:00.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:59.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:00.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:59.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:00.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:41:59.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:00.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:59.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:00.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:59.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:00.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:59.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:00.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:41:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:41:59.791+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:00.371 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:00.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:00.371 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:00.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:00.372 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:00.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:00.701 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:00.579+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 23928 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:00.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:00.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:00.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:00.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:00.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:00.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:00.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:00.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:00.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:00.814+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:01.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:01.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:01.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:01.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:01.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:01.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:01.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:01.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:01.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:02.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:02.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:02.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:02.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:02.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:01.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:01.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:01.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:01.862+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:03.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:02.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:02.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:02.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:02.855+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:02.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:02.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:03.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:02.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:02.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:02.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:02.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:03.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:03.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:03.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:03.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:03.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:03.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:03.862+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:04.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:04.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:04.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:05.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:04.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:04.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:04.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:04.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:04.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:04.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:04.864+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:05.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:05.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:05.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:05.870 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:05.583+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24000 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:05.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:05.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:05.871 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:05.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:05.871 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:05.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:06.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:06.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:06.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:06.015+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:05.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:05.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:05.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:05.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:07.061 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:06.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:07.061 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:06.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:07.061 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:06.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:07.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:06.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:07.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:06.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:07.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:06.859+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:07.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:06.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:07.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:07.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:07.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:08.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:07.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:08.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:07.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:08.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:07.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:08.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:07.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:08.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:07.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:08.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:07.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:08.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:07.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:08.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:08.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:08.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:08.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:08.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:08.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:08.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:08.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:08.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:08.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:09.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:09.097+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:09.097+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:09.097+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:10.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:09.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:10.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:09.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:10.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:09.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:10.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:09.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:10.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:09.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:10.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:09.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:10.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:09.886+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:10.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:10.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:10.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:10.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:10.900 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:10.587+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24072 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:10.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:10.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:10.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:10.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:10.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:10.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:11.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:11.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:11.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:11.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:10.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:10.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:10.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:10.900+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:12.076 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:11.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:12.076 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:11.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:12.077 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:11.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:12.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:11.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:12.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:11.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:12.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:11.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:12.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:11.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:12.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:12.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:12.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:12.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:12.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:12.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:12.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:12.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:12.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:12.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:12.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:12.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:12.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:13.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:13.041+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:13.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:13.041+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:13.041+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:13.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:13.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:13.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:13.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:13.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:13.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:13.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:13.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:13.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:14.343 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:14.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:14.344 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:14.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:14.344 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:14.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:14.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:14.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:14.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:14.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:14.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:14.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:14.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:14.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:15.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:15.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:15.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:15.920 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:15.590+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24144 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:15.921 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:15.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:15.921 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:15.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:15.921 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:15.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:16.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:15.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:16.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:15.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:15.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:15.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:16.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:16.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:16.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:16.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:16.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:16.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:16.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:16.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:16.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:16.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:16.955+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:17.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:17.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:17.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:17.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:17.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:17.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:17.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:17.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:17.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:17.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:18.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:18.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:18.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:19.070 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:18.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:19.071 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:18.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:19.071 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:18.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:19.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:18.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:19.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:18.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:19.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:18.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:19.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:18.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:19.386 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:19.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:19.386 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:19.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:19.386 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:19.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:20.032 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:19.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:20.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:19.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:20.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:19.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:20.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:19.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:20.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:19.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:20.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:19.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:20.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:19.899+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:20.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:20.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:20.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:20.890 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:20.593+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24216 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:20.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:20.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:20.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:20.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:20.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:20.715+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:21.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:21.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:21.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:21.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:21.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:20.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:20.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:20.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:21.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:20.889+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:21.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:21.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:21.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:21.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:21.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:21.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:21.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:21.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:21.855+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:22.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:22.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:22.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:22.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:22.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:22.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:22.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:22.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:22.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:22.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:23.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:23.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:23.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:23.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:23.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:23.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:23.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:23.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:23.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:23.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:24.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:24.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:24.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:24.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:24.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:24.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:24.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:24.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:24.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:24.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:24.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:24.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:24.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:24.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:24.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:24.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:24.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:24.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:24.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:24.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:25.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:25.072+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:25.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:25.072+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:25.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:25.072+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:25.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:25.597+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24288 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:25.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:25.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:25.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:25.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:25.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:25.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:25.761+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:26.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:26.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:26.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:26.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:26.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:26.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:26.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:26.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:26.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:26.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:26.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:26.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:26.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:26.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:27.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:27.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:27.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:27.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:27.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:27.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:27.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:27.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:27.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:27.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:27.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:28.142 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:27.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:28.143 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:27.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:28.143 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:27.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:28.143 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:27.755+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:28.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:28.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:28.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:28.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:28.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:28.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:28.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:28.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:28.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:28.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:28.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:28.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:28.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:28.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:28.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:29.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:29.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:29.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:29.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:29.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:29.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:29.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:29.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:29.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:30.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:29.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:30.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:29.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:30.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:29.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:30.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:29.741+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:30.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:30.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:30.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:30.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:30.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:30.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:30.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:30.600+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24360 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:30.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:30.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:30.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:30.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:30.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:30.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:30.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:30.766+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:31.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:31.196+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:31.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:31.196+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:31.196+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:31.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:31.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:31.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:31.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:31.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:31.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:31.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:31.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:31.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:32.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:32.244+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:32.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:32.244+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:32.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:32.244+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:32.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:32.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:32.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:32.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:32.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:32.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:32.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:33.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:33.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:33.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:33.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:33.574 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:33.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:33.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:33.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:33.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:33.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:33.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:33.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:34.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:33.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:34.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:33.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:34.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:33.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:34.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:33.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:34.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:34.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:34.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:34.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:34.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:34.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:34.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:34.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:34.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:34.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:34.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:34.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:34.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:34.818+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:35.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:35.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:35.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:35.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:35.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:35.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:35.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:35.604+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24431 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:35.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:35.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:35.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:35.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:35.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:35.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:35.825+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:36.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:36.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:36.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:36.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:36.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:36.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:36.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:36.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:36.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:36.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:36.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:36.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:36.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:36.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:36.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:36.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:37.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:37.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:37.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:37.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:37.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:37.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:37.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:37.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:37.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:37.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:37.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:37.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:37.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:37.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:37.805+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:38.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:38.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:38.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:38.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:38.852 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:38.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:38.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:38.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:38.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:38.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:39.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:38.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:39.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:38.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:39.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:38.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:39.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:38.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:39.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:39.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:39.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:39.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:39.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:39.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:39.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:39.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:39.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:39.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:39.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:39.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:40.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:39.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:40.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:39.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:40.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:39.868+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:40.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:39.868+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:40.260+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:40.260+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:40.260+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:40.799 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:40.607+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24503 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:40.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:40.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:40.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:40.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:40.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:40.557+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:40.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:40.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:40.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:41.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:40.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:41.559 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:41.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:41.559 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:41.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:41.559 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:41.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:41.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:41.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:41.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:41.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:41.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:41.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:42.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:41.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:42.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:41.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:42.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:41.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:42.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:41.812+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:42.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:42.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:42.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:42.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:42.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:42.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:42.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:42.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:42.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:42.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:42.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:42.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:42.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:42.819+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:43.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:43.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:43.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:43.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:43.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:43.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:43.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:43.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:43.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:43.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:43.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:43.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:43.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:43.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:44.592 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:44.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:44.593 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:44.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:44.593 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:44.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:44.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:44.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:44.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:44.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:44.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:44.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:44.778+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:44.778+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:44.778+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:44.778+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:45.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:45.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:45.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:45.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:45.605 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:45.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:45.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:45.612+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24574 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:45.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:45.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:45.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:45.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:45.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:45.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:45.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:46.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:46.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:46.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:46.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:46.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:46.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:46.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:46.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:46.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:47.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:46.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:46.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:46.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:46.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:47.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:47.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:47.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:47.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:47.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:47.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:47.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:47.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:47.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:47.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:47.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:47.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:47.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:48.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:48.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:48.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:48.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:48.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:48.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:48.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:48.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:48.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:48.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:48.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:48.643+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:48.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:48.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:48.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:48.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:48.715+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:49.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:49.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:49.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:49.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:49.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:49.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:50.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:49.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:50.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:49.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:49.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:49.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:50.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:50.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:50.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:50.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:50.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:50.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:50.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:50.616+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24647 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:50.646+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:50.646+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:50.646+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:50.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:50.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:50.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:50.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:51.502 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:51.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:51.503 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:51.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:51.503 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:51.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:51.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:51.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:51.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:51.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:51.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:51.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:51.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:51.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:51.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:51.708+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:52.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:52.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:52.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:52.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:52.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:52.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:52.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:52.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:52.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:52.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:52.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:52.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:52.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:52.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:52.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:52.659+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:53.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:53.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:53.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:53.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:53.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:53.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:53.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:53.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:53.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:53.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:54.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:54.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:54.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:54.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:54.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:54.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:54.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:54.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:54.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:54.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:54.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:54.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:54.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:55.686 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:42:55.619+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24718 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:42:55.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:55.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:55.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:55.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:55.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:55.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:55.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:55.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:55.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:55.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:55.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:55.790+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:55.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:55.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:55.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:55.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:56.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:56.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:56.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:56.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:56.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:56.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:56.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:56.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:56.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:56.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:56.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:56.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:56.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:56.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:56.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:56.802+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:56.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:56.802+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:56.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:56.802+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:57.763 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:57.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:57.763 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:57.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:57.763 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:57.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:57.763 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:57.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:57.763 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:57.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:57.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:57.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:57.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:57.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:58.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:57.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:58.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:57.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:58.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:57.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:58.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:58.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:58.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:58.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:58.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:58.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:58.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:58.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:59.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:58.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:59.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:58.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:59.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:58.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:59.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:59.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:42:59.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:42:59.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:59.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:59.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:59.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:59.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:59.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:42:59.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:42:59.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:59.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:59.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:59.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:59.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:42:59.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:42:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:42:59.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:00.583 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:00.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:00.584 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:00.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:00.584 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:00.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:00.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:00.623+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24791 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:00.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:00.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:00.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:00.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:00.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:00.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:00.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:01.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:01.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:01.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:01.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:01.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:01.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:01.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:01.546+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:01.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:01.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:01.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:01.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:01.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:01.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:02.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:02.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:02.688 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:02.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:02.688 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:02.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:02.529+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:02.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:02.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:02.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:02.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:02.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:02.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:02.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:02.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:02.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:03.666 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:03.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:03.666 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:03.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:03.667 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:03.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:03.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:03.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:03.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:03.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:03.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:03.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:03.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:03.488+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:03.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:03.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:03.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:04.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:04.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:04.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:04.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:04.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:04.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:04.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:04.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:04.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:04.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:04.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:04.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:04.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:04.509+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:04.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:04.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:04.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:04.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:04.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:04.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:05.627 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:05.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:05.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:05.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:05.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:05.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:05.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:05.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:05.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:05.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:05.628 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:05.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:05.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:05.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:05.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:05.626+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24861 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:05.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:05.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:05.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:05.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:06.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:06.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:06.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:06.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:06.671 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:06.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:06.671 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:06.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:06.671 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:06.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:06.671 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:06.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:06.671 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:06.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:06.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:06.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:06.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:06.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:06.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:06.669+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:07.709 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:07.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:07.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:07.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:07.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:07.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:07.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:07.426+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:07.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:07.426+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:07.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:07.426+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:07.710 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:07.427+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:07.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:07.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:07.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:07.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:07.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:08.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:08.322+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:08.681 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:08.322+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:08.681 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:08.322+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:08.681 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:08.456+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:08.681 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:08.456+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:08.681 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:08.456+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:08.681 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:08.456+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:08.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:08.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:08.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:09.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:09.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:09.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:09.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:09.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:09.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:09.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:09.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:09.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:09.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:09.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:09.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:09.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:09.479+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:09.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:09.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:09.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:09.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:09.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:09.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:10.631 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:10.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:10.631 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:10.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:10.631 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:10.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:10.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:10.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:10.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:10.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:10.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:10.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:10.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:10.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:10.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:10.630+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 24934 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:10.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:10.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:10.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:10.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:11.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:11.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:11.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:11.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:11.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:11.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:11.690 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:11.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:11.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:11.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:11.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:11.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:11.691 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:11.505+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:11.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:11.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:11.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:11.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:11.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:11.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:12.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:12.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:12.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:12.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:12.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:12.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:12.654 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:12.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:12.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:12.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:12.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:12.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:12.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:12.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:12.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:12.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:12.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:12.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:12.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:12.653+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:13.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:13.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:13.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:13.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:13.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:13.326+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:13.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:13.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:13.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:13.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:13.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:13.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:13.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:13.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:13.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:13.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:13.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:14.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:14.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:14.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:14.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:14.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:14.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:14.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:14.536+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:14.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:14.536+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:14.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:14.536+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:14.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:14.536+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:14.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:14.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:14.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:14.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:14.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:14.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:15.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:15.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:15.544 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:15.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:15.544 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:15.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:15.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:15.633+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25005 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:15.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:15.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:15.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:15.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:15.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:15.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:15.543+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:16.564 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:16.264+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:16.564 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:16.264+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:16.564 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:16.264+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:16.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:16.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:16.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:16.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:16.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:16.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:16.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:16.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:16.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:17.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:17.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:17.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:17.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:17.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:17.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:17.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:17.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:17.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:17.529+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:17.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:17.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:17.663+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:18.194+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:18.194+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:18.194+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:18.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:18.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:18.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:18.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:18.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:18.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:18.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:18.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:19.206+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:19.206+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:19.206+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:19.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:19.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:19.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:19.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:19.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:19.545+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:19.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:19.545+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:19.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:19.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:19.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:19.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:19.889 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:19.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:20.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:20.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:20.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:20.762 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:20.637+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25078 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:20.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:20.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:20.763 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:20.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:20.763 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:20.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:20.763 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:20.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:20.763 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:20.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:20.763 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:20.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:20.763 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:20.502+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:21.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:21.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:21.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:21.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:21.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:21.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:21.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:21.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:21.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:21.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:21.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:21.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:21.542+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:22.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:22.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:22.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:22.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:22.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:22.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:22.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:22.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:22.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:22.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:22.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:22.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:22.589+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:23.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:23.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:23.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:23.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:23.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:23.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:23.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:23.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:23.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:23.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:23.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:24.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:24.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:24.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:24.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:24.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:24.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:24.929 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:24.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:24.929 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:24.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:24.929 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:24.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:24.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:24.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:24.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:24.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:24.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:24.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:24.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:24.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:25.545 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:25.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:25.545 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:25.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:25.545 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:25.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:25.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:25.641+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25149 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:25.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:25.752+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:25.752+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:25.752+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:25.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:25.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:25.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:25.544+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:26.501 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:26.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:26.502 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:26.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:26.502 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:26.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:26.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:26.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:26.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:26.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:26.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:26.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:26.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:26.501+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:27.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:27.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:27.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:27.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:27.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:27.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:27.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:27.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:27.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:27.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:27.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:27.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:27.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:27.477+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:27.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:27.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:27.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:28.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:28.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:28.279+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:28.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:28.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:28.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:28.459+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:28.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:28.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:28.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:29.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:29.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:29.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:29.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:29.432+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:29.432+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:29.432+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:29.432+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:29.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:29.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:29.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:30.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:30.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:30.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:30.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:30.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:30.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:30.647 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:30.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:30.647 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:30.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:30.647 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:30.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:30.647 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:30.451+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:30.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:30.645+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25222 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:30.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:30.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:30.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:30.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:30.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:30.778+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:31.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:31.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:31.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:31.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:31.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:31.451+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:31.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:31.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:31.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:31.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:31.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:31.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:31.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:32.531 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:32.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:32.531 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:32.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:32.531 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:32.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:32.532 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:32.480+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:32.532 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:32.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:32.532 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:32.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:32.532 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:32.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:32.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:32.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:32.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:33.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:33.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:33.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:33.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:33.668 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:33.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:33.668 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:33.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:33.669 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:33.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:33.669 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:33.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:33.669 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:33.442+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:33.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:33.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:33.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:33.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:33.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:33.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:34.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:34.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:34.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:34.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:34.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:34.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:34.441+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:35.013 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:34.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:35.013 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:34.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:35.013 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:34.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:35.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:35.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:35.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:35.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:35.444 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:35.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:35.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:35.649+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25293 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:35.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:35.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:35.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:35.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:35.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:35.444+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:36.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:35.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:36.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:35.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:36.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:35.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:36.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:36.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:36.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:36.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:36.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:36.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:36.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:36.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:36.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:36.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:36.452+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:37.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:36.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:37.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:36.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:37.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:36.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:37.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:37.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:37.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:37.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:37.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:37.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:37.432+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:37.432+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:37.432+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:37.432+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:38.221 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:37.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:38.221 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:37.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:38.221 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:37.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:38.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:38.220+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:38.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:38.221+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:38.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:43:38.221+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:03.866491+0000 (oldest deadline 2024-02-19T19:33:23.866491+0000) 2024-02-19T19:43:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:38.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:38.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:38.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:38.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:38.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:38.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:43:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:43:38.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:05.004460+0000 (oldest deadline 2024-02-19T19:33:25.004460+0000) 2024-02-19T19:43:39.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:39.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:39.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:39.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:39.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:39.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:39.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:39.474+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:40.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:40.653+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25366 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:40.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:40.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:40.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:40.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:40.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:40.452+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:41.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:41.424+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:41.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:41.424+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:41.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:41.424+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:41.424+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:42.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:42.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:42.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:42.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:42.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:42.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:42.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:43.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:43.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:43.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:43.489+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:44.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:44.515+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:44.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:44.515+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:44.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:44.515+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:44.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:44.515+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:45.781 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:45.657+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25437 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:45.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:45.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:45.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:45.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:45.492+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:46.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:46.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:46.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:46.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:46.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:46.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:46.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:46.523+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:47.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:47.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:47.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:47.537+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:48.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:48.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:48.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:48.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:48.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:48.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:48.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:49.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:49.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:49.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:49.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:49.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:49.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:49.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:50.963 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:50.661+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25510 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:50.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:50.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:50.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:50.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:50.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:50.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:50.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:50.627+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:51.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:51.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:51.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:33:15.051297+0000 (oldest deadline 2024-02-19T19:33:35.051297+0000) 2024-02-19T19:43:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:51.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:52.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:52.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:53.659+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:54.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:54.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:55.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:43:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:43:55.663+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25581 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:43:55.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:55.679+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:56.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:56.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:57.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:58.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:43:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:43:59.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:00.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:00.667+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25654 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:00.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:01.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:01.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:01.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:01.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:01.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:01.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:01.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:01.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:01.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:02.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:02.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:02.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:02.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:02.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:02.520+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:02.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:02.520+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:02.520+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:03.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:03.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:03.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:03.573+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:03.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:03.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:03.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:03.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:03.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:04.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:04.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:04.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:04.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:04.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:04.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:04.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:04.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:04.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:04.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:04.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:04.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:05.470 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:05.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:05.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:05.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:05.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:05.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:05.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:05.671+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25725 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:05.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:05.584+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:05.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:05.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:05.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:05.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:05.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:06.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:06.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:06.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:06.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:06.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:06.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:06.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:06.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:06.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:06.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:06.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:07.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:07.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:07.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:07.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:07.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:07.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:07.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:07.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:07.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:07.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:08.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:08.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:08.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:08.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:08.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:08.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:08.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:09.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:09.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:09.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:09.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:09.549+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:09.549+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:09.549+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:10.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:10.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:10.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:10.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:10.675+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25798 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:10.598+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:10.583+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:10.583+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:10.583+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:11.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:11.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:11.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:11.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:11.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:11.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:11.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:11.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:12.467 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:12.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:12.467 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:12.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:12.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:12.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:12.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:12.584+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:12.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:12.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:12.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:12.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:13.094+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:13.094+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:13.094+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:13.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:13.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:13.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:13.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:14.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:14.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:14.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:14.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:14.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:14.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:14.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:14.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:14.592+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:14.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:14.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:14.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:14.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:14.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:15.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:15.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:15.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:15.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:15.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:15.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:15.679 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:15.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:15.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:15.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:15.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:15.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:15.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:15.678+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25869 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:15.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:15.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:15.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:15.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:15.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:15.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:15.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:16.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:16.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:16.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:16.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:16.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:16.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:16.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:16.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:16.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:16.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:17.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:17.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:17.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:17.594+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:17.594+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:17.594+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:17.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:17.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:17.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:17.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:18.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:18.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:18.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:18.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:18.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:18.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:18.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:18.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:18.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:18.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:19.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:19.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:19.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:19.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:19.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:19.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:19.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:19.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:19.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:19.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:20.390 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:20.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:20.390 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:20.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:20.390 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:20.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:20.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:20.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:20.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:20.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:20.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:20.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:20.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:20.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:20.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:20.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:20.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:20.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:20.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:20.564+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:20.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:20.683+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 25942 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:21.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:21.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:21.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:21.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:21.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:21.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:21.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:21.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:21.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:21.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:21.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:21.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:21.602+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:22.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:22.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:22.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:22.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:22.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:22.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:22.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:22.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:22.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:22.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:23.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:23.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:23.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:23.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:23.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:23.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:23.535+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:23.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:23.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:23.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:24.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:24.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:24.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:24.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:24.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:24.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:24.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:24.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:24.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:24.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:24.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:24.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:24.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:24.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:25.431 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:25.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:25.431 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:25.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:25.431 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:25.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:25.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:25.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:25.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:25.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:25.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:25.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:25.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:25.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:25.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:25.687+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26013 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:25.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:25.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:25.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:26.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:26.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:26.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:26.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:26.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:26.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:26.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:26.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:26.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:26.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:26.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:26.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:26.581+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:27.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:27.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:27.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:27.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:27.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:27.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:27.553+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:27.553+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:27.553+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:27.553+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:28.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:28.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:28.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:28.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:28.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:28.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:28.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:28.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:28.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:28.558+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:29.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:29.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:29.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:29.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:29.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:29.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:29.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:29.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:29.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:29.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:29.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:29.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:29.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:30.473 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:30.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:30.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:30.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:30.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:30.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:30.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:30.691+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26086 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:30.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:30.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:30.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:30.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:30.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:30.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:30.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:30.616+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:30.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:30.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:30.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:31.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:31.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:31.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:31.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:31.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:31.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:31.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:31.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:31.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:31.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:32.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:32.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:32.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:32.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:32.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:32.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:32.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:33.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:32.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:32.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:32.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:33.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:33.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:33.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:33.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:33.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:33.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:33.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:33.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:33.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:33.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:33.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:33.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:33.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:33.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:34.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:33.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:34.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:33.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:34.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:33.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:34.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:34.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:34.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:34.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:34.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:34.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:34.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:34.551+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:34.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:34.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:34.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:34.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:34.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:34.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:34.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:34.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:34.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:35.818 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:35.694+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26157 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:35.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:35.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:35.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:35.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:35.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:35.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:35.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:35.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:35.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:35.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:35.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:35.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:35.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:35.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:35.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:35.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:35.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:36.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:36.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:36.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:36.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:36.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:36.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:36.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:36.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:36.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:36.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:36.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:36.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:36.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:36.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:36.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:36.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:36.943+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:37.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:37.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:37.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:37.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:37.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:37.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:37.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:37.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:37.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:37.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:37.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:37.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:37.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:37.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:37.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:37.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:37.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:38.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:38.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:38.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:38.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:38.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:38.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:38.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:38.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:38.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:38.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:38.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:39.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:39.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:39.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:39.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:39.683+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:39.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:39.683+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:39.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:39.683+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:39.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:39.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:39.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:39.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:39.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:39.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:39.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:39.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:39.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:39.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:39.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:40.824 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:40.698+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26230 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:40.824 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:40.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:40.824 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:40.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:40.824 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:40.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:40.825 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:40.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:40.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:40.655+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:40.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:40.655+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:40.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:40.655+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:41.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:40.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:41.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:40.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:41.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:40.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:41.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:41.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:41.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:41.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:41.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:41.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:41.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:41.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:41.928 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:41.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:41.929 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:41.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:41.929 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:41.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:41.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:41.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:41.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:42.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:42.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:42.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:42.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:42.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:42.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:42.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:42.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:42.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:42.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:42.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:42.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:42.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:42.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:42.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:42.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:42.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:43.938 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:43.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:43.939 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:43.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:43.939 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:43.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:43.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:43.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:43.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:43.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:43.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:43.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:43.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:43.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:44.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:43.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:44.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:43.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:44.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:43.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:44.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:44.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:44.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:44.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:44.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:44.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:44.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:44.745+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:44.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:44.745+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:44.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:44.745+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:44.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:44.745+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:44.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:44.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:44.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:45.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:45.701+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26301 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:45.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:45.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:45.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:45.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:45.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:45.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:45.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:45.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:45.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:46.259 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:45.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:46.259 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:45.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:46.259 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:45.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:46.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:46.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:46.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:46.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:46.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:46.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:46.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:46.721+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:46.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:46.721+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:46.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:46.721+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:47.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:47.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:47.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:47.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:47.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:47.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:47.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:47.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:47.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:47.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:47.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:49.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:48.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:48.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:48.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:48.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:48.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:48.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:48.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:48.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:48.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:48.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:49.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:49.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:49.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:49.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:49.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:49.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:49.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:49.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:49.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:49.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:50.921 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:50.704+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26374 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:50.921 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:50.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.921 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:50.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.921 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:50.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:50.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:50.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:50.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:50.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:50.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:50.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:50.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:50.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:51.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:50.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:51.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:50.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:51.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:50.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:52.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:51.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:52.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:51.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:52.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:51.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:51.762+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:51.762+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:51.762+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:52.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:51.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:52.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:51.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:51.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:51.754+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:53.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:52.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:53.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:52.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:53.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:52.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:52.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:52.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:52.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:52.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:52.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:53.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:52.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:53.232 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:52.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:53.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:53.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:53.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:53.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:53.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:53.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:53.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:53.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:53.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:53.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:53.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:53.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:53.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:53.774+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:54.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:53.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:54.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:53.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:54.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:53.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:54.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:54.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:54.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:54.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:54.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:54.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:54.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:54.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:54.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:54.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:55.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:44:55.708+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26445 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:44:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:55.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:55.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:55.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:55.694+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:56.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:55.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:56.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:55.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:56.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:55.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:56.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:56.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:56.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:56.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:56.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:56.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:56.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:56.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:56.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:56.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:56.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:56.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:56.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:57.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:57.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:57.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:57.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:57.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:57.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:57.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:57.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:58.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:58.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:58.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:59.028 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:58.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:59.028 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:58.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:59.028 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:58.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:44:59.028 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:58.729+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:59.028 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:58.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:59.029 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:58.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:59.029 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:58.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:44:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:59.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:59.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:44:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:44:59.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:00.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:59.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:00.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:59.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:00.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:59.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:00.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:44:59.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:00.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:59.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:00.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:59.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:00.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:44:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:44:59.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:00.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:00.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:00.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:00.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:00.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:00.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:00.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:00.712+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26518 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:00.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:00.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:00.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:00.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:00.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:00.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:00.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:00.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:00.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:01.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:01.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:01.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:01.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:01.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:01.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:01.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:01.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:01.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:01.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:02.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:02.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:02.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:03.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:02.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:03.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:02.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:03.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:02.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:03.033 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:02.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:03.034 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:02.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:03.034 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:02.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:03.034 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:02.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:03.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:03.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:03.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:03.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:03.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:03.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:03.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:03.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:03.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:03.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:04.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:04.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:04.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:04.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:04.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:04.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:04.801+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:04.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:04.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:04.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:04.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:04.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:05.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:05.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:05.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:05.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:05.716+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26589 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:05.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:05.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:05.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:05.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:05.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:05.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:05.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:05.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:05.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:05.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:06.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:06.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:06.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:07.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:06.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:07.076 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:06.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:07.076 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:06.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:07.076 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:06.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:07.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:06.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:07.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:06.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:07.077 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:06.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:07.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:07.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:07.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:08.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:07.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:08.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:07.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:08.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:07.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:08.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:07.831+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:08.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:07.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:08.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:07.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:08.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:07.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:08.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:08.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:08.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:08.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:08.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:08.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:08.835+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:08.835+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:08.835+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:08.835+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:09.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:09.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:09.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:09.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:09.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:09.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:10.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:09.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:10.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:09.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:10.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:09.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:10.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:09.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:10.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:09.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:10.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:09.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:10.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:09.865+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:10.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:10.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:10.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:10.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:10.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:10.721+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26662 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:10.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:10.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:10.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:10.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:10.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:10.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:10.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:10.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:10.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:10.914+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:11.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:11.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:11.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:12.016 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:11.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:12.016 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:11.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:12.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:11.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:12.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:11.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:12.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:11.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:12.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:11.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:12.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:11.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:12.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:12.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:12.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:12.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:12.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:12.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:12.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:12.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:12.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:12.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:12.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:12.928+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:13.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:13.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:13.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:13.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:13.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:13.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:13.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:13.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:13.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:13.893+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:13.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:13.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:13.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:14.975 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:14.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:14.975 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:14.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:14.975 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:14.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:14.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:14.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:14.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:14.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:14.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:14.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:14.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:14.894+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:15.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:14.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:15.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:14.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:15.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:14.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:15.918 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:15.725+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26733 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:15.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:15.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:15.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:15.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:15.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:15.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:16.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:15.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:16.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:15.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:16.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:15.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:16.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:15.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:15.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:15.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:15.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:16.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:16.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:16.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:16.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:16.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:16.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:17.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:16.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:17.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:16.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:17.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:16.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:17.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:16.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:16.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:16.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:16.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:17.930 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:17.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:17.930 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:17.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:17.931 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:17.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:18.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:18.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:18.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:18.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:18.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:18.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:17.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:17.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:17.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:17.930+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:18.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:18.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:18.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:18.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:18.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:18.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:19.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:18.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:19.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:18.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:19.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:18.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:19.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:18.898+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:19.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:18.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:19.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:18.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:18.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:19.908 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:19.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:19.909 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:19.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:19.909 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:19.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:20.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:19.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:20.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:19.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:20.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:19.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:20.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:19.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:20.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:19.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:20.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:19.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:20.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:19.909+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:20.894 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:20.729+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26806 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:20.894 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:20.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:20.894 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:20.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:20.894 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:20.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:21.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:21.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:21.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:21.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:20.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:20.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:20.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:21.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:20.953+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:21.907 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:21.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:21.908 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:21.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:21.908 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:21.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:22.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:22.036+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:22.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:22.036+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:22.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:22.036+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:22.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:21.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:22.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:21.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:21.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:21.907+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:22.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:22.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:22.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:22.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:22.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:22.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:22.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:22.880+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:23.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:23.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:23.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:23.925 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:23.622+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:23.926 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:23.622+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:23.926 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:23.622+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:24.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:24.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:24.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:24.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:24.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:24.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:24.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:23.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:23.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:23.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:23.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:24.901 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:24.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:24.901 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:24.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:24.901 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:24.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:25.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:25.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:25.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:25.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:25.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:25.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:25.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:24.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:25.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:24.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:25.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:24.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:25.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:24.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:25.935 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:25.733+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26877 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:25.662+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:25.662+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:25.662+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:25.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:25.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:25.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:25.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:25.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:26.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:26.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:26.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:26.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:26.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:26.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:26.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:26.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:26.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:26.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:26.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:26.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:26.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:26.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:26.873+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:27.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:27.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:27.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:27.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:27.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:27.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:27.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:27.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:27.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:27.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:27.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:27.846+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:28.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:28.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:28.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:28.894 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:28.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:28.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:28.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:28.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:28.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:29.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:29.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:29.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:29.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:28.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:28.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:28.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:28.894+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:29.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:29.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:29.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:29.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:29.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:29.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:30.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:30.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:30.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:30.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:30.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:29.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:30.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:29.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:30.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:29.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:30.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:29.894+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:30.935 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:30.736+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 26950 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:30.935 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:30.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:30.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:30.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:30.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:30.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:30.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:30.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:30.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:31.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:30.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:30.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:30.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:30.935+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:31.902 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:31.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:31.902 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:31.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:31.903 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:31.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:31.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:31.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:31.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:31.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:31.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:31.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:31.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:32.931 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:32.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:32.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:32.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:32.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:32.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:32.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:32.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:32.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:32.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:32.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:32.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:32.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:32.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:33.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:32.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:32.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:32.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:33.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:33.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:33.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:33.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:33.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:33.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:34.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:33.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:34.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:33.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:34.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:33.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:34.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:33.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:34.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:33.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:34.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:33.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:33.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:34.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:34.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:34.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:34.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:34.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:34.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:34.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:34.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:34.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:34.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:34.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:34.944+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:34.945+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:35.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:35.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:35.621+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:36.018 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:35.739+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27021 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:36.018 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:35.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:36.019 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:35.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:36.019 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:35.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:36.019 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:35.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:36.019 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:35.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:36.019 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:35.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:36.019 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:35.905+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:36.862 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:36.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:36.862 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:36.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:36.862 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:36.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:36.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:36.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:36.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:36.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:36.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:36.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:36.861+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:37.935 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:37.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:37.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:37.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:37.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:37.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:37.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:37.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:37.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:37.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:37.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:37.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:37.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:37.850+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:37.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:37.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:37.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:38.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:38.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:38.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:38.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:38.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:38.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:38.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:38.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:38.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:39.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:38.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:38.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:38.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:38.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:39.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:39.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:39.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:39.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:39.869 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:39.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:39.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:39.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:39.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:40.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:39.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:39.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:39.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:39.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:40.869 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:40.743+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27088 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:40.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:40.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:40.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:40.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:40.870 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:40.617+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:40.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:40.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:40.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:41.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:40.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:41.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:41.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:41.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:41.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:41.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:41.899 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:41.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:41.899 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:41.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:41.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:41.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:41.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:42.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:41.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:42.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:41.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:42.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:41.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:41.898+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:42.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:42.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:42.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:42.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:42.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:42.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:42.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:42.903+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:43.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:42.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:43.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:42.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:43.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:42.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:43.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:43.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:43.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:43.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:43.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:43.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:43.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:43.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:43.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:43.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:43.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:43.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:43.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:43.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:44.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:43.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:44.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:43.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:44.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:43.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:44.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:44.693+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:44.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:44.693+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:44.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:44.693+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:45.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:44.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:44.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:44.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:44.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:44.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:44.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:44.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:45.895 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:45.747+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27153 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:45.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:45.644+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:45.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:45.644+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:45.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:45.644+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:46.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:45.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:46.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:45.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:46.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:45.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:46.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:45.915+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:46.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:45.895+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:45.895+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:45.895+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:46.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:46.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:46.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:46.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:46.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:46.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:46.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:46.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:46.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:46.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:46.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:46.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:47.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:46.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:47.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:46.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:47.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:46.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:47.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:46.936+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:47.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:47.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:47.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:47.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:47.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:47.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:47.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:47.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:47.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:47.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:47.959 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:47.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:48.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:47.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:48.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:47.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:48.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:47.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:48.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:47.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:49.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:48.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:49.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:48.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:49.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:48.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:49.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:48.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:49.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:48.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:48.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:48.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:48.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:48.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:48.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:50.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:49.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:50.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:49.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:50.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:49.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:50.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:49.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:50.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:49.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:49.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:49.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:49.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:49.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:49.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:51.145 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:50.751+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27220 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:51.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:50.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:51.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:50.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:51.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:50.837+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:51.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:50.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:51.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:50.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:51.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:50.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:51.146 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:50.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:51.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:50.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:51.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:50.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:51.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:50.937+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:52.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:51.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:52.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:51.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:52.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:51.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:52.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:51.931+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:51.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:51.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:51.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:51.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:51.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:51.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:52.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:52.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:52.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:52.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:52.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:52.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:52.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:52.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:52.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:52.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:53.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:53.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:53.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:53.845+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:53.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:53.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:53.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:53.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:53.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:53.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:53.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:55.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:54.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:55.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:54.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:55.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:54.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:55.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:54.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:55.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:54.780+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:54.780+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:54.780+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:54.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:54.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:54.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:56.186 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:45:55.755+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27282 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:45:56.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:55.876+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:56.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:55.876+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:56.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:55.876+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:56.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:55.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:56.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:55.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:56.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:55.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:56.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:55.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:56.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:55.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:56.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:55.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:56.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:55.848+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:57.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:56.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:57.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:56.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:57.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:56.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:57.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:56.840+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:56.840+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:56.840+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:57.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:56.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:57.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:56.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:57.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:56.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:57.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:56.855+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:58.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:57.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:58.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:57.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:58.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:57.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:57.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:57.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:57.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:57.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:57.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:57.860+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:57.860+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:59.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:58.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:59.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:58.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:59.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:58.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:45:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:58.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:58.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:58.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:45:59.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:58.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:58.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:58.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:45:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:58.841+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:00.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:59.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:00.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:59.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:00.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:45:59.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:00.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:59.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:59.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:45:59.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:00.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:59.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:00.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:59.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:00.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:59.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:00.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:45:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:45:59.816+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:01.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:00.761+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27345 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:01.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:00.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:01.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:00.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:01.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:00.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:01.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:00.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:00.806+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:00.806+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:00.806+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:00.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:00.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:00.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:02.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:01.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:02.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:01.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:02.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:01.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:01.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:01.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:01.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:02.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:01.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:02.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:01.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:02.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:01.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:01.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:03.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:02.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:03.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:02.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:03.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:02.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:03.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:02.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:03.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:02.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:03.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:02.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:03.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:02.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:03.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:02.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:03.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:02.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:03.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:02.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:03.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:03.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:03.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:03.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:03.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:03.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:03.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:03.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:03.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:03.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:03.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:05.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:04.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:05.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:04.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:05.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:04.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:04.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:04.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:04.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:04.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:04.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:04.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:04.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:06.128 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:05.764+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27414 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:06.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:05.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:06.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:05.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:06.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:05.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:06.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:05.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:06.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:05.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:06.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:05.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:06.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:05.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:06.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:05.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:06.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:05.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:06.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:05.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:06.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:06.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:06.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:06.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:06.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:06.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:06.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:06.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:06.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:06.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:06.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:06.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:06.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:06.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:06.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:06.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:06.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:06.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:06.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:07.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:07.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:07.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:07.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:07.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:07.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:07.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:07.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:07.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:07.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:07.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:07.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:07.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:08.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:08.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:08.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:08.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:08.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:08.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:08.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:08.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:08.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:08.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:08.771+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:09.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:09.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:09.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:09.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:09.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:09.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:09.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:09.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:09.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:09.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:09.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:09.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:09.774+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:10.892 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:10.769+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27483 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:10.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:10.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:10.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:10.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:10.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:10.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:10.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:10.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:10.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:10.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:10.893 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:10.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:10.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:10.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:10.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:10.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:10.894 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:10.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:10.894 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:10.824+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:11.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:11.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:11.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:11.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:11.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:11.660+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:11.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:11.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:11.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:11.819+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:12.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:12.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:12.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:12.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:12.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:12.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:12.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:12.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:12.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:12.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:12.768+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:12.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:12.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:12.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:13.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:13.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:13.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:13.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:13.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:13.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:13.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:13.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:13.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:13.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:13.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:13.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:13.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:14.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:14.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:14.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:14.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:14.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:14.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:14.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:14.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:14.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:14.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:14.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:14.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:14.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:14.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:14.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:14.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:15.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:15.773+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27558 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:15.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:15.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:15.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:15.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:15.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:15.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:15.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:15.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:15.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:15.812+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:16.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:16.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:16.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:16.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:16.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:16.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:16.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:16.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:16.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:16.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:16.766+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:17.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:17.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:17.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:17.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:17.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:17.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:17.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:17.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:17.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:17.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:18.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:18.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:18.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:18.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:18.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:18.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:18.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:18.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:18.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:18.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:18.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:19.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:19.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:19.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:19.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:19.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:19.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:19.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:19.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:19.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:19.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:19.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:19.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:19.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:20.904 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:20.777+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27630 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:20.904 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:20.629+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:20.904 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:20.629+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:20.904 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:20.629+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:20.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:20.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:20.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:20.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:20.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:20.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:20.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:20.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:20.905 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:20.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:20.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:20.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:20.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:20.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:21.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:21.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:21.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:21.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:21.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:21.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:21.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:21.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:21.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:21.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:21.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:21.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:21.716+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:21.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:21.716+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:21.716+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:22.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:22.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:22.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:22.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:22.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:22.645+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:22.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:22.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:22.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:22.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:22.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:22.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:22.766+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:23.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:23.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:23.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:23.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:23.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:23.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:23.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:23.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:23.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:23.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:23.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:23.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:23.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:24.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:24.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:24.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:24.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:24.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:24.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:24.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:24.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:24.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:24.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:24.766+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:24.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:24.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:24.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:24.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:25.976 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:25.780+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27702 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:25.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:25.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:25.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:25.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:25.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:25.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:25.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:25.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:25.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:25.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:25.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:25.664+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:26.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:26.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:26.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:26.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:26.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:26.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:26.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:26.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:26.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:26.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:26.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:26.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:26.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:26.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:26.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:26.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:26.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:26.776+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:27.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:27.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:27.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:27.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:27.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:27.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:27.705+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:27.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:27.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:27.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:27.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:28.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:28.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:28.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:28.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:28.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:28.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:28.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:28.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:28.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:28.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:28.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:28.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:28.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:29.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:29.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:29.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:29.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:29.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:29.618+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:29.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:29.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:29.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:29.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:29.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:29.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:29.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:30.912 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:30.785+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27774 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:30.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:30.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:30.913 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:30.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:30.913 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:30.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:30.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:30.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:30.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:30.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:30.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:30.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:30.913 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:30.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:30.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:30.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:30.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:30.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:30.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:30.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:31.745 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:31.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:31.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:31.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:31.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:31.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:31.746 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:31.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:31.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:31.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:31.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:31.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:31.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:31.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:32.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:31.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:31.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:31.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:32.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:32.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:32.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:32.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:32.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:32.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:32.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:32.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:32.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:32.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:32.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:32.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:33.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:33.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:33.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:33.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:33.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:33.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:33.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:33.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:33.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:33.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:33.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:33.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:33.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:34.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:34.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:34.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:34.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:34.853 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:34.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:34.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:34.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:34.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:34.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:34.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:34.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:34.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:34.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:35.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:34.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:34.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:34.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:35.790 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:35.790 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:35.790 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:35.791 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:35.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:35.791 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:35.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:35.791 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:35.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:35.791 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:35.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:36.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:35.789+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27846 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:35.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:35.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:35.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:36.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:36.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:36.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:36.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:36.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:36.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:36.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:36.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:36.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:36.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:37.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:36.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:37.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:36.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:37.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:36.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:37.811 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:37.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:37.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:37.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:37.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:37.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:37.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:37.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:37.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:37.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:37.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:37.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:37.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:37.675+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:38.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:37.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:38.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:37.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:38.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:37.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:38.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:38.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:38.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:38.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:38.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:38.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:38.841 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:38.841 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:38.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:38.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:38.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:39.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:38.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:39.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:38.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:39.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:38.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:39.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:39.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:39.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:39.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:39.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:39.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:39.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:39.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:39.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:39.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:39.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:39.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:39.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:39.670+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:40.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:39.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:40.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:39.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:40.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:39.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:40.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:40.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:40.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:40.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:40.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:40.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:40.795 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:40.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:40.795 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:40.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:40.795 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:40.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:40.795 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:40.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:41.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:40.793+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27918 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:41.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:40.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:41.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:40.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:41.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:40.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:41.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:41.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:41.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:41.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:41.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:41.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:41.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:41.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:41.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:41.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:41.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:41.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:41.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:41.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:42.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:41.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:42.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:41.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:42.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:41.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:42.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:42.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:42.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:42.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:42.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:42.570+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:42.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:42.570+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:42.570+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:42.570+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:43.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:42.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:43.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:42.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:43.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:42.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:43.681 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:43.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:43.682 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:43.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:43.682 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:43.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:43.682 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:43.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:43.682 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:43.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:43.682 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:43.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:43.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:43.600+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:44.385 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:43.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:44.385 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:43.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:44.385 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:43.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:44.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:44.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:44.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:44.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:44.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:44.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:44.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:44.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:44.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:44.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:44.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:44.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:44.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:44.572+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:45.370 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:45.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:45.370 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:45.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:45.371 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:45.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:45.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:45.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:45.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:45.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:45.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:45.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:45.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:45.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:45.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:45.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:45.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:45.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:45.609+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:46.053 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:45.797+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 27989 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:46.397 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:46.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:46.397 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:46.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:46.397 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:46.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:46.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:46.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:46.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:46.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:46.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:46.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:46.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:46.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:46.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:46.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:46.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:46.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:46.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:46.593+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:47.371 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:47.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:47.372 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:47.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:47.372 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:47.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:47.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:47.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:47.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:47.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:47.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:47.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:47.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:47.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:47.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:47.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:48.414 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:48.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:48.414 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:48.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:48.414 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:48.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:48.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:48.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:48.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:48.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:48.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:48.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:48.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:48.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:48.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:48.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:48.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:48.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:48.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:48.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:49.430 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:49.078+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:49.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:49.078+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:49.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:49.078+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:49.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:49.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:49.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:49.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:49.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:49.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:49.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:49.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:49.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:49.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:49.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:50.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:50.055+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:50.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:50.055+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:50.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:50.055+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:50.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:50.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:50.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:50.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:50.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:50.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:50.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:50.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:50.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:50.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:50.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:50.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:50.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:51.067 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:50.802+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28060 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:51.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:51.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:51.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:51.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:51.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:51.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:51.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:51.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:51.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:51.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:51.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:51.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:51.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:51.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:52.440 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:52.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:52.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:52.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:52.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:52.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:52.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:52.441+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:52.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:52.441+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:52.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:52.441+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:52.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:52.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:52.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:52.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:52.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:52.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:52.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:52.661+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:53.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:53.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:53.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:53.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:53.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:53.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:53.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:53.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:53.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:53.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:53.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:53.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:53.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:53.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:53.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:53.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:53.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:53.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:54.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:54.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:54.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:54.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:54.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:54.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:54.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:54.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:54.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:54.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:54.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:54.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:54.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:54.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:54.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:54.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:54.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:54.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:54.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:54.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:55.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:55.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:55.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:55.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:55.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:55.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:55.806 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:55.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:55.806 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:55.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:55.806 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:55.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:55.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:55.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:55.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:55.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:55.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:55.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:55.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:55.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:56.107 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:46:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:46:55.806+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28132 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:46:56.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:56.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:56.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:56.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:56.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:56.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:56.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:56.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:56.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:56.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:56.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:56.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:56.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:56.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:56.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:56.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:56.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:56.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:57.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:57.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:57.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:57.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:57.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:57.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:57.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:57.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:57.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:57.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:57.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:57.571+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:57.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:57.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:57.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:57.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:58.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:58.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:58.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:58.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:58.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:58.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:58.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:58.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:58.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:58.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:58.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:58.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:58.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:59.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:59.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:59.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:46:59.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:46:59.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:59.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:59.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:59.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:59.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:46:59.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:46:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:59.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:59.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:59.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:46:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:46:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:46:59.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:00.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:00.119+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:00.119+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:00.119+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:00.809 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:00.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:00.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:00.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:00.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:00.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:00.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:00.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:00.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:00.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:00.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:00.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:00.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:00.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:01.116 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:00.810+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28203 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:01.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:01.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:01.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:01.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:01.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:01.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:01.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:01.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:01.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:01.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:01.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:01.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:02.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:01.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:02.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:01.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:02.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:01.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:02.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:01.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:02.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:02.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:02.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:02.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:02.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:02.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:02.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:02.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:02.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:02.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:02.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:02.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:02.802+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:02.802+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:02.802+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:02.802+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:03.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:03.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:03.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:03.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:03.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:03.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:03.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:03.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:03.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:03.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:03.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:03.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:03.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:03.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:03.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:03.830+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:04.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:04.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:04.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:04.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:04.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:04.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:04.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:04.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:04.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:04.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:05.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:05.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:05.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:05.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:04.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:04.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:04.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:04.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:05.881 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:05.814+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28274 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:05.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:05.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:05.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:05.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:05.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:05.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:06.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:06.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:06.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:06.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:06.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:06.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:05.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:05.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:05.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:05.882+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:06.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:06.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:06.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:06.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:06.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:06.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:07.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:07.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:07.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:07.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:07.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:07.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:07.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:06.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:07.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:06.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:07.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:06.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:07.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:06.879+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:07.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:07.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:07.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:07.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:07.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:07.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:08.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:08.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:08.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:08.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:08.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:08.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:07.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:07.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:07.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:07.898+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:08.906 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:08.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:08.906 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:08.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:08.906 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:08.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:09.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:09.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:09.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:09.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:08.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:08.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:08.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:08.906+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:09.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:09.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:09.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:09.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:09.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:09.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:10.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:10.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:10.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:10.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:10.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:10.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:10.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:09.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:10.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:09.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:10.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:09.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:10.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:09.934+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:10.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:10.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:10.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:10.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:10.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:10.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:10.944 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:10.817+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28346 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:11.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:11.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:11.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:11.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:11.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:11.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:11.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:10.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:11.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:10.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:11.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:10.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:11.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:10.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:11.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:11.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:11.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:11.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:11.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:11.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:12.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:11.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:12.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:11.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:12.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:11.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:12.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:11.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:12.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:12.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:12.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:12.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:12.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:12.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:12.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:12.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:12.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:12.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:12.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:12.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:13.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:13.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:13.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:13.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:13.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:13.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:13.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:12.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:13.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:12.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:13.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:12.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:13.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:12.942+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:13.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:13.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:13.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:13.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:13.959 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:13.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:14.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:14.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:14.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:14.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:14.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:14.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:14.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:13.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:14.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:13.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:14.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:13.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:14.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:13.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:14.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:14.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:14.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:14.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:14.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:14.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:15.086+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:15.086+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:15.086+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.003+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:15.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:15.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:15.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:15.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:15.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:15.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:16.134 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:15.821+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28418 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:16.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:16.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:16.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:16.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:15.992+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:16.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:16.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:16.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:16.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:16.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:16.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:16.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:16.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:16.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:16.574+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:17.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:17.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:17.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:17.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:17.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:17.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:17.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:17.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:17.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:17.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:17.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:17.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:17.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:17.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:17.608+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:18.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:18.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:18.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:18.115+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:18.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:18.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:18.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:18.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:18.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:18.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:18.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:18.008+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:18.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:18.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:18.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:18.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:18.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:18.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:19.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:19.121+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:19.121+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:19.121+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:19.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:19.054+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:19.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:19.054+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:19.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:19.054+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:19.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:19.054+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:19.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:19.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:19.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:19.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:19.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:19.616+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:20.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:20.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:20.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:20.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:20.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:20.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:20.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:20.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:20.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:20.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:20.095+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:20.837 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:20.825+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28490 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:20.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:20.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:20.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:20.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:20.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:20.577+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:21.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:21.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:21.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:21.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:21.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:21.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:21.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:21.067+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:21.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:21.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:21.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:21.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:21.899 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:21.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:21.899 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:21.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:21.899 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:21.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:22.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:22.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:22.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:22.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:22.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:22.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:22.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:22.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:22.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:22.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:22.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:22.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:22.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:22.085+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:22.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:22.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:22.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:22.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:22.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:22.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:23.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:23.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:23.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:23.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:23.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:23.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:23.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:23.042+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:23.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:23.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:23.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:23.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:23.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:23.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:23.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:23.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:23.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:23.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:23.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:23.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:24.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:24.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:24.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:24.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:24.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:24.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:24.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:24.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:24.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:24.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:24.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:24.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:24.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:24.060+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:24.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:24.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:24.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:24.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:24.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:24.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:25.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:25.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:25.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:25.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:25.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:25.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:25.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:25.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:25.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:25.046+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:25.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:25.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:25.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:25.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:25.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:25.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:26.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:25.829+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28562 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:26.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:26.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:26.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:26.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:26.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:26.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:26.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:26.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:26.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:26.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:26.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:26.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:26.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:26.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:26.811 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:26.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:26.811 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:26.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:26.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:26.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:27.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:27.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:27.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:27.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:27.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:27.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:27.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:27.059+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:27.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:27.059+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:27.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:27.059+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:27.060+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:27.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:27.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:27.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:27.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:27.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:28.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:28.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:28.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:28.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:28.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:28.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:28.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:28.066+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:28.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:28.066+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:28.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:28.066+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:28.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:28.066+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:28.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:28.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:28.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:28.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:29.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:28.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:29.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:28.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:29.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:28.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:29.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:29.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:29.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:29.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:29.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:29.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:29.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:29.042+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:29.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:29.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:29.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:29.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:29.465+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:30.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:30.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:30.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:30.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:30.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:30.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:30.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:30.057+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:30.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:30.034+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:30.034+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:30.034+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:30.834 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:30.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:30.834 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:30.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:30.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:30.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:31.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:30.833+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28635 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:31.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:31.045+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:31.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:31.045+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:31.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:31.045+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:31.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:31.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:31.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:31.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:31.043+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:31.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:31.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:31.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:31.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:31.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:31.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:32.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:32.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:32.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:32.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:32.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:32.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:32.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:32.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:32.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:32.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:32.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:32.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:32.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:32.057+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:32.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:32.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:32.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:32.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:32.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:33.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:33.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:33.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:33.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:33.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:33.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:33.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:33.076+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:33.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:33.076+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:33.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:33.076+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:33.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:33.076+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:33.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:33.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:33.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:33.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:33.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:33.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:34.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:34.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:34.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:34.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:34.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:34.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:34.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:34.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:34.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:34.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:34.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:34.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:34.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:34.117+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:34.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:34.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:34.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:34.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:34.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:34.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:35.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:35.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:35.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:35.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:35.449 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:35.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:35.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:35.103+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:35.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:35.103+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:35.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:35.103+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:35.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:35.103+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:35.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:35.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:35.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:35.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:35.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:35.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:36.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:35.838+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28708 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:36.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:36.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:36.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:36.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:36.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:36.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:36.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:36.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:36.084+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:36.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:36.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:36.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:36.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:36.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:36.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:37.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:37.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:37.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:37.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:37.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:37.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:37.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:37.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:37.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:37.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:37.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:37.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:37.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:37.089+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:37.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:37.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:37.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:37.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:37.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:37.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:38.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:37.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:38.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:37.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:38.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:37.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:38.116+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:38.116+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:38.116+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:38.116+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:38.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:38.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:38.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:38.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:38.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:38.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:39.209 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:38.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:39.210 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:38.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:39.210 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:38.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:39.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:39.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:39.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:39.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:39.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:39.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:39.210 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:39.130+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:39.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:39.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:39.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:39.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:40.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:39.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:40.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:39.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:40.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:39.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:40.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:40.149+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:40.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:40.149+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:40.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:40.149+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:40.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:40.149+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:40.368 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:40.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:40.369 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:40.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:40.369 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:40.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:41.130 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:40.842+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28781 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:41.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:40.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:41.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:40.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:41.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:40.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:41.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:41.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:41.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:41.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:41.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:41.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:41.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:41.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:41.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:41.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:41.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:41.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:41.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:41.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:41.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:41.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:42.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:42.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:42.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:42.124+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:42.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:42.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:42.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:42.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:43.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:42.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:43.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:42.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:43.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:42.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:43.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:43.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:43.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:43.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:43.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:43.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:43.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:43.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:43.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:43.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:43.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:43.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:43.687 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:43.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:43.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:43.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:43.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:43.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:43.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:43.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:44.370 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:44.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:44.371 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:44.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:44.371 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:44.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:44.371 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:44.111+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:44.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:44.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:44.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:44.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:44.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:44.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:45.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:44.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:45.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:44.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:45.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:44.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:45.109+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:45.109+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:45.109+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:45.109+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:45.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:45.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:45.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:45.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:45.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:45.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:46.122 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:45.846+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28854 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:46.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:45.950+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:46.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:45.950+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:46.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:45.950+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:46.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:46.122+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:46.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:46.122+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:46.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:46.122+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:46.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:46.123+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:46.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:46.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:46.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:46.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:46.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:47.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:46.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:47.109 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:46.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:47.109 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:46.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:47.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:47.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:47.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:47.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:47.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:47.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:47.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:47.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:47.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:47.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:47.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:47.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:47.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:47.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:48.268 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:47.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:47.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:47.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:48.109+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:48.109+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:48.270 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:48.109+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:48.270 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:48.109+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:48.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:48.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:48.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:48.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:48.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:48.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:49.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:49.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:49.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:49.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:49.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:49.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:49.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:49.071+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:49.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:48.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:49.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:48.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:49.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:48.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:49.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:49.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:49.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:49.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:49.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:50.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:49.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:50.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:49.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:50.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:49.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:50.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:50.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:50.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:50.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:50.071+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:50.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:50.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:50.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:50.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:50.484 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:50.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:51.079 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:50.850+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 28927 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:51.079 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:50.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:51.079 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:50.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:51.080 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:50.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:51.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:51.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:51.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:51.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:51.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:51.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:51.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:51.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:51.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:51.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:51.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:51.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:51.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:51.081+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:52.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:51.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:52.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:51.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:52.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:51.976+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:52.382 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:52.229+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:52.382 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:52.229+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:52.382 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:52.229+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:52.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:52.102+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:52.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:52.102+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:52.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:52.102+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:52.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:52.102+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:53.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:52.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:53.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:52.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:53.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:52.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:53.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:53.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:53.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:53.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:53.072+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:53.687 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:53.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:53.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:53.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:53.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:53.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:54.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:54.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:54.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:54.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:54.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:54.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:54.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:54.042+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:53.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:53.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:53.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:54.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:54.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:54.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:54.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:54.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:54.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:55.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:55.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:55.299 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:55.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:55.299 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:55.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:55.299 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:55.087+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:55.299 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:54.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:55.299 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:54.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:55.299 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:54.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:55.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:55.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:55.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:55.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:55.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:55.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:56.124 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:47:55.854+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29000 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:47:56.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:55.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:56.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:55.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:56.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:55.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:56.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:56.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:56.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:56.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:56.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:56.325+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:56.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:56.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:56.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:56.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:56.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:56.124+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:57.326 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:57.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:57.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:57.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:57.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:57.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:57.327 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:57.086+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:57.327 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:56.987+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:57.327 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:56.987+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:57.327 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:56.987+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:57.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:57.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:57.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:57.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:57.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:57.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:58.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:58.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:58.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:58.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:58.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:58.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:58.360 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:58.041+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:58.360 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:58.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:58.360 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:58.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:58.360 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:58.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:58.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:58.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:58.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:58.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:58.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:58.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:59.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:59.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:59.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:47:59.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:59.346 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:59.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:59.346 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:59.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:59.346 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:47:59.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:47:59.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:59.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:59.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:59.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:47:59.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:47:59.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:00.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:00.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:00.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:00.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:00.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:00.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:00.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:00.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:00.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.961+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:00.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:47:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:47:59.961+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:00.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:00.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:00.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:00.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:00.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:00.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:01.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:00.857+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29073 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:01.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:01.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:01.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:01.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:01.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:01.050+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:01.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:01.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:01.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:01.003+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:01.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:01.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:01.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:01.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:01.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:01.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:02.323 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:02.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:02.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:02.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:02.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:02.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:02.324 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:02.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:02.324 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:02.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:02.324 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:02.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:02.325 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:02.001+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:02.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:02.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:02.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:02.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:02.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:02.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:03.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:03.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:03.282 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:03.283 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:03.283 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:03.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:03.283 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:03.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:03.283 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:03.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:03.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:03.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:03.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:03.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:03.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:03.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:04.256 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:04.256 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:04.256 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:04.257 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:03.980+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:04.257 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:04.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:04.257 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:04.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:04.257 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:04.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:04.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:04.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:04.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:04.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:04.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:05.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:04.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:05.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:04.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:04.937+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:04.937+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:05.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:05.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:05.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:05.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:05.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:05.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:05.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:05.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:05.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:06.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:05.861+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29145 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:05.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:05.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:05.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:06.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:05.954+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:06.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:06.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:06.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:06.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:06.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:06.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:06.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:06.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:06.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:07.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:06.911+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:07.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:06.911+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:07.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:06.911+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:07.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:06.911+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:07.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:07.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:07.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:07.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:07.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:07.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:07.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:07.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:07.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:07.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:08.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:07.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:08.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:08.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:08.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:08.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:08.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:08.286+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:08.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:08.286+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:08.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:08.286+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:09.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:08.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:08.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:08.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:09.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:08.886+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:09.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:09.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:09.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:09.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:09.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:09.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:09.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:09.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:09.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:09.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:10.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:09.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:10.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:09.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:10.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:09.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:10.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:09.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:10.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:10.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:10.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:10.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:10.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:10.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:10.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:10.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:10.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:10.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:10.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:10.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:11.144 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:10.866+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29217 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:11.145 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:10.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:11.145 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:10.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:11.145 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:10.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:11.145 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:10.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:11.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:11.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:11.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:11.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:11.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:11.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:11.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:11.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:11.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:11.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:11.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:12.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:11.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:12.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:11.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:12.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:11.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:12.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:11.928+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:12.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:12.170+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:12.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:12.170+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:12.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:12.170+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:12.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:12.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:12.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:12.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:12.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:12.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:12.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:12.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:12.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:12.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:13.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:13.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:13.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:13.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:13.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:13.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:13.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:13.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:13.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:13.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:13.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:13.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:14.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:14.172+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:14.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:14.172+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:14.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:14.172+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:14.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:14.130+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:14.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:14.130+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:14.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:14.130+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:13.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:13.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:13.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:13.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:15.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:15.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:15.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:15.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:15.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:15.098+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:15.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:14.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:15.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:14.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:15.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:14.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:15.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:14.934+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:15.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:15.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:15.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:15.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:15.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:16.147 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:15.870+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29290 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:16.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:15.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:16.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:15.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:16.148 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:15.914+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:16.148 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:15.914+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:16.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:16.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:16.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:16.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:16.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:16.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:16.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:16.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:16.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:17.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:16.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:17.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:16.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:17.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:16.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:17.159 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:16.907+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:17.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:17.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:17.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:17.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:17.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:17.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:17.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:18.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:18.143+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:18.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:18.143+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:18.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:18.143+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:18.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:18.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:18.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:18.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:18.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:18.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:17.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:17.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:17.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:18.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:17.928+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:19.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:19.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:19.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:19.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:19.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:19.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:19.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:19.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:19.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:19.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:19.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:19.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:18.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:18.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:18.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:18.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:20.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:20.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:20.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:20.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:20.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:20.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:20.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:19.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:20.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:19.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:20.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:19.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:20.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:19.922+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:20.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:20.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:20.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:21.140 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:20.875+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29362 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:21.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:21.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:21.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:21.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:21.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:21.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:21.141 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:20.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:21.142 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:20.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:21.142 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:20.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:21.142 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:20.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:21.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:21.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:21.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:21.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:21.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:22.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:22.172+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:22.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:22.172+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:22.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:22.172+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:22.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:22.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:22.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:21.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:21.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:21.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:22.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:21.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:23.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:23.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:23.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:23.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:23.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:23.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:23.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:22.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:22.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:22.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:22.992+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:24.370 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:24.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:24.371 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:24.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:24.371 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:24.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:24.371 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:24.209+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:24.371 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:24.209+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:24.372 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:24.209+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:24.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:24.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:24.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:24.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:24.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:24.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:24.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:24.040+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:25.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:25.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:25.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:25.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:25.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:25.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:25.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:25.002+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:25.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:25.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:25.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:25.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:25.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:25.172+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:25.172+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:25.172+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:26.166 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:25.881+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29435 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:26.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:26.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:26.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:26.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:26.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:26.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:26.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:26.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:26.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:26.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.026+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:26.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:26.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:26.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:26.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:26.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:26.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:27.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:27.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:27.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:27.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:27.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:27.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:27.153 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:27.153 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:27.153 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:27.153 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:26.981+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:27.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:27.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:27.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:27.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:27.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:27.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:28.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:28.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:28.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:28.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:28.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:28.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:28.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:28.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:28.185+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:27.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:27.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:27.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:27.942+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:29.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:29.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:29.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:29.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:29.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:29.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:29.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:28.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:29.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:28.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:29.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:28.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:29.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:28.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:29.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:29.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:29.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:29.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:29.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:29.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:29.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:29.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:29.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:29.929+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:30.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:30.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:30.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:30.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:30.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:30.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:30.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:30.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:30.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:30.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:30.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:31.191 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:30.885+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29507 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:31.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:31.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:31.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:31.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:31.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:31.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:31.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:30.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:31.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:30.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:31.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:30.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:31.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:30.923+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:31.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:31.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:31.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:31.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:31.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:31.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:32.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:32.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:32.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:32.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:31.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:31.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:31.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:31.954+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:32.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:32.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:32.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:33.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:32.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:33.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:32.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:33.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:32.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:33.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:32.906+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:33.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:33.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:33.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:33.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:33.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:33.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:33.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:33.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:33.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:33.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:33.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:33.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:33.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:33.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:33.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:33.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:33.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:34.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:34.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:34.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:34.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:34.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:34.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:34.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:34.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:34.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:34.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:34.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:34.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:35.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:35.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:35.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:35.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:35.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:34.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:34.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:34.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:34.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:35.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:35.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:35.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:35.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:36.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:35.889+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29580 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:36.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:36.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:36.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:36.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:35.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:36.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:35.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:36.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:35.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:35.803+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:36.658 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:36.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:36.658 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:36.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:36.659 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:36.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:36.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:36.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:36.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:36.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:36.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:36.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:36.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:36.802+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:37.310 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:37.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:37.310 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:37.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:37.310 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:37.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:37.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:37.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:37.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:37.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:37.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:37.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:38.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:38.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:38.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:38.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:37.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:37.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:37.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:37.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:38.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:38.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:38.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:38.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:38.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:38.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:39.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:38.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:39.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:38.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:39.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:38.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:39.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:38.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:39.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:39.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:39.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:39.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:39.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:39.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:39.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:39.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:39.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:40.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:39.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:40.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:39.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:40.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:39.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:40.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:39.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:40.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:40.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:40.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:40.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:40.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:40.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:40.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:41.124 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:40.893+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29653 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:41.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:40.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:41.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:40.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:41.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:40.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:41.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:40.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:41.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:41.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:41.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:41.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:41.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:41.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:41.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:41.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:41.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:42.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:41.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:42.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:41.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:42.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:41.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:42.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:41.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:42.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:42.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:42.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:42.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:42.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:42.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:42.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:42.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:42.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:42.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:42.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:42.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:42.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:42.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:42.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:42.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:42.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:42.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:42.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:42.728+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:43.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:43.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:43.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:43.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:43.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:43.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:43.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:43.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:43.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:43.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:43.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:43.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:43.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:43.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:43.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:43.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:43.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:43.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:44.359 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:44.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:44.359 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:44.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:44.359 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:44.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:44.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:44.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:44.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:44.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:44.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:44.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:44.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:44.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:44.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:44.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:44.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:44.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:44.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:44.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:45.268 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:45.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:45.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:45.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:45.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:45.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:45.671 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:45.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:45.671 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:45.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:45.671 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:45.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:45.976 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:45.896+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29726 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:45.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:45.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:45.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:45.670+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:46.344 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:46.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:46.344 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:46.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:46.344 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:46.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:46.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:46.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:46.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:46.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:46.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:46.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:46.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:46.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:46.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:46.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:46.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:46.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:47.361 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:46.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:47.361 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:46.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:47.362 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:46.992+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:47.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:47.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:47.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:47.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:47.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:47.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:47.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:47.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:47.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:47.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:47.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:47.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:48.390 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:48.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:48.390 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:48.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:48.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:48.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:48.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:48.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:48.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:48.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:48.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:48.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:48.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:48.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:48.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:48.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:48.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:48.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:48.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:48.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:49.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:48.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:49.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:48.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:49.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:48.965+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:49.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:49.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:49.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:49.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:49.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:49.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:49.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:49.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:49.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:49.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:49.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:49.547+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:50.255 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:49.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:50.255 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:49.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:50.255 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:49.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:50.568 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:50.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:50.568 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:50.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:50.568 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:50.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:50.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:50.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:50.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:50.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:50.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:50.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:50.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:50.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:51.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:50.901+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29799 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:51.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:51.004+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:51.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:51.004+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:51.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:51.004+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:51.726 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:51.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:51.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:51.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:51.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:51.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:51.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:51.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:51.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:51.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:51.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:51.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:51.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:51.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:52.347 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:52.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:52.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:52.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:52.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:52.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:52.347+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:52.347+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:52.347+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:52.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:52.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:52.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:52.650 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:52.557+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:53.327 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:53.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:53.328 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:53.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:53.328 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:53.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:53.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:53.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:53.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:53.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:53.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:53.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:53.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:53.556+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:53.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:53.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:53.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:53.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:53.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:53.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:54.351 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:54.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:54.351 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:54.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:54.351 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:54.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:54.726 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:54.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:54.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:54.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:54.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:54.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:54.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:54.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:54.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:54.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:54.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:54.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:54.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:54.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:55.394 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:55.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:55.395 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:55.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:55.395 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:55.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:55.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:55.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:55.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:55.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:55.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:55.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:55.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:55.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:55.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:55.557+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:56.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:48:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:48:55.906+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29872 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:48:56.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:56.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:56.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:56.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:56.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:56.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:56.726 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:56.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:56.726 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:56.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:56.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:56.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:56.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:56.553+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:56.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:56.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:56.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:56.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:56.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:56.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:57.406 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:57.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:57.406 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:57.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:57.407 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:57.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:57.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:57.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:57.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:57.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:57.692 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:57.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:57.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:57.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:57.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:57.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:57.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:57.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:57.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:57.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:58.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:58.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:58.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:58.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:58.433 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:58.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:58.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:58.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:58.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:58.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:58.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:58.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:58.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:58.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:58.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:58.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:58.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:59.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:59.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:59.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:48:59.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:48:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:59.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:59.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:59.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:48:59.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:48:59.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:59.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:59.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:59.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:48:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:48:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:48:59.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:00.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:00.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:00.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:00.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:00.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:00.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:00.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:00.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:00.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:00.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:00.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:00.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:00.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:00.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:00.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:01.208 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:00.910+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 29945 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:01.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:01.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:01.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:01.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:01.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:01.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:01.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:01.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:01.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:01.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:01.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:01.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:01.670+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:01.670+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:02.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:02.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:02.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:02.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:02.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:02.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:02.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:02.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:02.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:02.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:02.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:02.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:02.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:02.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:02.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:02.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:02.706+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:03.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:03.444+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:03.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:03.444+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:03.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:03.444+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:03.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:03.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:03.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:03.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:03.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:03.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:03.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:03.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:04.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:04.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:04.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:04.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:04.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:04.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:04.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:04.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:04.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:04.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:04.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:04.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:05.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:05.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:05.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:05.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:05.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:05.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:05.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:05.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:05.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:05.937 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:05.915+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30018 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:05.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:05.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:05.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:05.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:05.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:05.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:05.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:05.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:06.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:06.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:06.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:06.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:06.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:06.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:06.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:06.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:06.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:06.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:06.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:06.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:06.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:06.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:06.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:06.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:06.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:07.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:07.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:07.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:07.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:07.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:07.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:07.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:07.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:07.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:07.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:07.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:08.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:08.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:08.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:08.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:08.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:08.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:08.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:08.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:08.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:08.636+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:09.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:09.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:09.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:09.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:09.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:09.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:09.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:09.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:09.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:09.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:09.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:09.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:09.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:09.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:10.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:10.151+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:10.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:10.151+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:10.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:10.151+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:10.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:10.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:10.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:10.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:10.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:10.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:10.726 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:10.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:10.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:10.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:10.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:10.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:10.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:10.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:11.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:10.920+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30091 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:11.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:11.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:11.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:11.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:11.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:11.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:11.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:11.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:11.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:11.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:11.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:11.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:11.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:11.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:11.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:11.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:11.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:11.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:11.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:11.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:12.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:12.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:12.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:12.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:12.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:12.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:12.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:12.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:12.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:13.221 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:12.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:13.221 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:12.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:13.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:13.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:13.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:13.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:13.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:13.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:13.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:13.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:13.220+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:13.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:13.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:13.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:13.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:13.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:14.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:14.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:14.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:14.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:14.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:14.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:14.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:14.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:14.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:15.206 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:14.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:15.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:14.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:15.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:14.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:15.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:14.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:15.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:15.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:15.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:15.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:15.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:15.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:15.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:16.047 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:15.926+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30163 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:16.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:15.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:16.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:15.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:16.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:15.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:16.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:15.791+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:16.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:16.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:16.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:16.262+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:16.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:16.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:16.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:16.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:16.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:16.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:16.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:16.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:16.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:16.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:16.820+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:17.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:17.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:17.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:17.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:17.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:17.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:17.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:18.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:17.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:18.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:17.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:18.141 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:17.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:18.141 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:17.833+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:18.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:18.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:18.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:18.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:18.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:18.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:18.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:19.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:18.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:19.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:18.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:19.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:18.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:19.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:18.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:19.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:19.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:19.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:19.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:19.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:19.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:20.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:19.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:20.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:19.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:20.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:19.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:20.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:19.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:20.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:20.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:20.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:20.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:20.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:20.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:21.056 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:20.930+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30237 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:21.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:20.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:21.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:20.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:21.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:20.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:21.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:20.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:21.360 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:21.165+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:21.360 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:21.165+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:21.361 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:21.165+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:21.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:21.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:21.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:21.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:21.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:21.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:22.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:21.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:22.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:21.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:22.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:21.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:22.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:21.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:22.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:22.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:22.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:22.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:22.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:22.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:22.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:23.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:22.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:23.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:22.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:23.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:22.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:23.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:22.748+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:23.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:23.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:23.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:23.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:23.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:23.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:23.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:23.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:23.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:23.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:23.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:23.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:23.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:24.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:24.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:24.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:24.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:24.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:24.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:25.153 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:24.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:25.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:24.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:25.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:24.800+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:25.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:24.800+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:25.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:25.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:25.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:25.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:25.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:25.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:25.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:25.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:25.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:25.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:25.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:25.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:26.061 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:25.934+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30309 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:26.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:25.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:26.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:25.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:26.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:25.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:26.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:25.822+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:26.422 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:26.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:26.422 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:26.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:26.422 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:26.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:26.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:26.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:26.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:26.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:26.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:26.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:27.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:26.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:27.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:26.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:27.147 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:26.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:27.148 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:26.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:27.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:27.147+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:27.402 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:27.147+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:27.402 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:27.147+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:27.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:27.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:27.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:27.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:27.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:27.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:28.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:28.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:28.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:28.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:28.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:28.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:28.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:27.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:28.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:27.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:27.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:28.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:27.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:28.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:28.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:28.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:28.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:29.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:29.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:29.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:29.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:29.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:28.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:29.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:28.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:28.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:28.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:29.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:29.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:29.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:29.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:30.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:30.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:30.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:30.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:30.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:29.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:30.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:29.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:30.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:29.873+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:30.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:29.873+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:30.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:30.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:30.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:30.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:31.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:30.938+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30383 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:31.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:31.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:31.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:31.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:30.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:30.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:30.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:30.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:31.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:31.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:31.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:31.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:31.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:31.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:32.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:32.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:32.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:32.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:32.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:31.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:31.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:31.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:31.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:32.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:32.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:32.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:32.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:32.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:32.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:33.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:33.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:33.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:33.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:33.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:32.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:33.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:32.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:32.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:32.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:33.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:33.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:33.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:33.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:33.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:33.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:33.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:33.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:33.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:33.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:33.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:33.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:33.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:33.826+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:34.417 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:33.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:34.417 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:33.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:34.418 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:33.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:34.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:34.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:34.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:34.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:35.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:35.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:35.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:35.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:35.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:34.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:34.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:34.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:34.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:35.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:35.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:35.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:35.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:36.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:35.942+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30455 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:36.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:36.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:36.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:36.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:35.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:35.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:35.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:35.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:36.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:36.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:36.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:36.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:36.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:36.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:37.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:37.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:37.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:37.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:36.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:36.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:36.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:36.903+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:37.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:37.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:37.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:37.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:37.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:37.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:38.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:38.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:38.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:37.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:38.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:37.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:37.915+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:37.915+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:38.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:38.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:38.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:38.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:38.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:38.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:38.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:38.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:38.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:39.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:38.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:38.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:38.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:38.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:39.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:39.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:39.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:39.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:39.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:39.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:40.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:40.021+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:40.021+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:40.021+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:40.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:39.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:40.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:39.921+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:39.921+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:39.921+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:40.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:40.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:40.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:40.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:40.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:40.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:41.171 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:40.947+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30529 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:41.171 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:41.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:41.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:41.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:41.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:41.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:41.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:40.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:41.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:40.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:41.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:40.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:41.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:40.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:41.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:41.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:41.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:42.036+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:42.036+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:42.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:42.036+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:41.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:41.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:41.908+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:41.908+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:42.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:42.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:42.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:42.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:43.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:43.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:43.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:42.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:42.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:42.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:42.886+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:43.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:43.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:43.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:43.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:43.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:43.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:43.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:43.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:43.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:43.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:43.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:44.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:44.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:44.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:44.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:44.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:44.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:44.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:44.276+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:44.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:44.276+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:44.276+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:45.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:45.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:45.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:45.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:45.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:44.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:44.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:44.861+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:44.862+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:45.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:45.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:45.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:45.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:45.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:45.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:46.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:45.951+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30601 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:46.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:46.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:46.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:46.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:46.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:46.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:45.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:45.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:45.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:45.889+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:46.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:46.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:46.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:47.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:46.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:47.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:46.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:47.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:46.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:47.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:46.845+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:47.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:47.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:47.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:47.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:47.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:47.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:47.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:47.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:48.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:47.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:48.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:47.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:48.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:47.798+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:48.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:47.798+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:48.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:48.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:48.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:48.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:48.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:48.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:48.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:49.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:48.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:49.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:48.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:49.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:48.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:49.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:48.815+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:49.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:49.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:49.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:49.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:49.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:49.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:49.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:49.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:49.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:49.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:50.153 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:49.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:50.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:49.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:50.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:49.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:50.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:49.833+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:50.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:50.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:50.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:50.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:50.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:50.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:50.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:51.081 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:50.955+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30675 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:51.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:50.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:51.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:50.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:51.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:50.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:51.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:50.813+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:51.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:51.166+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:51.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:51.166+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:51.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:51.166+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:51.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:51.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:51.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:52.138 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:51.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:52.138 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:51.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:52.138 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:51.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:52.138 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:51.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:52.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:52.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:52.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:52.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:52.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:52.195+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:52.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:52.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:52.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:52.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:53.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:52.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:53.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:52.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:53.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:52.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:53.152 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:52.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:53.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:53.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:53.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:53.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:53.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:53.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:53.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:53.170+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:53.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:53.170+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:53.170+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:53.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:53.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:53.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:53.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:54.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:54.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:54.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:54.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:54.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:54.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:54.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:54.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:54.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:54.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:54.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:54.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:54.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:54.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:54.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:54.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:54.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:54.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:55.689 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:55.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:55.689 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:55.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:55.689 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:55.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:55.689 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:55.228+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:55.689 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:55.228+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:55.690 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:55.228+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:55.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:55.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:55.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:55.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:55.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:55.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:55.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:55.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:56.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:49:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:49:55.959+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30743 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:49:56.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:56.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:56.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:56.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:56.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:56.531 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:56.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:56.531 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:56.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:56.531 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:56.268+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:56.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:56.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:56.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:56.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:56.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:56.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:57.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:57.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:57.182+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:57.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:57.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:57.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:57.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:57.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:57.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:57.598+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:58.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:58.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:58.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:58.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:58.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:58.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:58.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:58.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:58.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:58.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:58.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:58.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:59.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:59.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:49:59.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:49:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:59.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:59.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:49:59.336+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:49:59.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:59.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:59.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:59.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:49:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:49:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:49:59.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:00.378 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:00.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:00.379 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:00.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:00.379 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:00.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:00.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:00.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:00.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:00.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:00.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:00.378+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:00.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:00.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:00.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:00.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:00.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:00.598+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:01.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:00.964+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30817 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:01.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:01.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:01.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:01.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:01.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:01.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:01.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:01.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:01.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:01.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:01.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:01.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:01.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:01.584+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:01.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:01.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:01.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:02.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:02.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:02.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:02.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:02.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:02.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:02.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:02.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:02.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:02.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:02.590+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:03.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:03.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:03.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:03.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:03.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:03.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:03.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:03.542+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:03.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:03.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:03.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:04.526 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:04.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:04.526 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:04.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:04.526 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:04.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:04.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:04.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:04.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:04.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:04.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:04.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:04.590+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:05.558 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:05.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:05.558 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:05.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:05.558 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:05.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:05.559 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:05.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:05.559 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:05.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:05.559 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:05.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:05.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:05.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:05.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:05.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:05.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:05.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:05.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:05.558+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:06.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:05.970+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30889 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:06.524 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:06.292+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:06.525 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:06.292+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:06.525 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:06.292+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:06.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:06.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:06.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:06.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:06.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:06.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:06.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:06.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:06.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:06.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:06.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:06.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:06.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:06.577+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:07.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:07.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:07.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:07.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:07.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:07.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:07.583 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:07.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:07.584 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:07.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:07.584 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:07.519+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:07.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:07.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:07.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:07.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:07.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:08.530 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:08.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:08.530 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:08.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:08.530 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:08.311+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:08.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:08.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:08.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:08.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:08.588+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:08.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:08.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:08.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:09.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:09.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:09.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:09.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:09.591 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:09.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:09.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:09.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:09.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:09.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:09.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:09.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:09.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:09.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:09.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:09.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:09.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:10.616 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:10.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:10.616 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:10.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:10.616 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:10.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:10.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:10.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:10.617 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:10.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:10.617 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:10.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:10.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:10.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:10.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:10.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:10.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:10.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:10.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:10.616+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:11.226 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:10.975+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 30963 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:11.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:11.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:11.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:11.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:11.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:11.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:11.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:11.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:11.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:11.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:11.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:11.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:11.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:11.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:12.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:12.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:12.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:12.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:12.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:12.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:12.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:12.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:12.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:12.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:12.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:12.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:12.430+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:13.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:13.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:13.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:13.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:13.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:13.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:13.603 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:13.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:13.604 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:13.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:13.604 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:13.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:13.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:13.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:13.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:13.603+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:14.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:14.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:14.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:14.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:14.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:14.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:14.964 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:14.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:14.964 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:14.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:14.964 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:14.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:14.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:14.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:14.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:14.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:14.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:14.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:14.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:14.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:15.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:15.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:15.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:15.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:15.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:15.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:15.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:15.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:15.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:15.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:15.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:15.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:15.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:16.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:15.979+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31034 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:16.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:16.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:16.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:16.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:16.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:16.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:16.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:16.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:16.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:16.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:16.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:16.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:16.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:16.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:16.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:16.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:16.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:17.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:17.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:17.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:17.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:17.564+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:17.564+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:17.564+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:17.661+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:17.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:17.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:17.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:18.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:18.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:18.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:18.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:18.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:18.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:18.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:18.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:18.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:18.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:18.694 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:18.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:18.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:18.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:18.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:18.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:18.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:18.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:18.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:18.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:19.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:19.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:19.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:19.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:19.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:19.598+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:19.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:19.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:19.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:19.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:20.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:20.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:20.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:20.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:20.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:20.426+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:20.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:20.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:20.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:20.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:20.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:20.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:20.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:20.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:20.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:20.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:20.694+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:21.385 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:20.984+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31101 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:21.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:21.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:21.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:21.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:21.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:21.590+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:21.590+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:21.590+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:21.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:21.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:21.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:22.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:21.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:22.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:22.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:22.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:22.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:22.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:22.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:22.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:22.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:22.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:22.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:23.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:22.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:23.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:22.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:23.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:22.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:23.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:22.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:23.620 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:23.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:23.621 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:23.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:23.621 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:23.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:23.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:23.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:23.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:23.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:23.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:23.619+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:23.619+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:23.619+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:24.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:24.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:24.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:24.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:24.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:24.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:24.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:24.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:24.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:24.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:24.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:24.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:24.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:24.801+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:25.661 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:25.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:25.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:25.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:25.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:25.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:25.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:25.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:25.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:25.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:25.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:25.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:25.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:25.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:25.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:25.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:25.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:26.313 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:25.987+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31172 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:26.675 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:26.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:26.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:26.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:26.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:26.312+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:26.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:26.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:26.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:26.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:26.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:26.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:26.729+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:27.692 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:27.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:27.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:27.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:27.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:27.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:27.692+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:27.692+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:27.692+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:27.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:27.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:27.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:27.729+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:28.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:28.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:28.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:29.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:28.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:29.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:28.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:29.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:28.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:28.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:28.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:28.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:28.776+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:29.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:29.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:29.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:30.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:29.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:30.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:29.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:30.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:29.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:29.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:29.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:29.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:30.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:29.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:30.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:30.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:30.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:30.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:30.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:30.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:31.119 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:30.992+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31246 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:31.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:30.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:31.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:30.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:31.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:30.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:31.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:30.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:31.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:30.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:31.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:30.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:31.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:30.814+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:31.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:31.266+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:31.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:31.266+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:31.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:31.266+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:31.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:31.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:31.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:31.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:31.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:31.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:31.824+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:32.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:32.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:32.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:32.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:32.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:32.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:33.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:32.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:33.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:32.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:33.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:32.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:32.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:32.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:32.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:32.814+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:33.699 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:33.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:33.700 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:33.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:33.700 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:33.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:33.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:33.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:33.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:33.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:33.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:33.811+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:33.811+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:34.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:34.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:34.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:34.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:34.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:34.251+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:34.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:34.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:34.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:34.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:34.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:34.654+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:34.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:34.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:34.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:34.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:35.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:35.211+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:35.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:35.211+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:35.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:35.211+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:35.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:35.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:35.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:35.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:35.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:35.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:35.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:35.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:35.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:35.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:35.857+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:36.467 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:35.997+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31318 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:36.467 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:36.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:36.467 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:36.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:36.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:36.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:36.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:36.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:36.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:36.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:36.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:36.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:36.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:36.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:36.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:37.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:36.817+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:37.685 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:37.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:37.685 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:37.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:37.685 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:37.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:37.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:37.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:37.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:37.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:37.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:37.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:37.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:38.528 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:38.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:38.529 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:38.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:38.529 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:38.260+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:38.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:38.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:38.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:38.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:38.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:38.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:38.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:39.661 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:39.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:39.661 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:39.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:39.661 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:39.252+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:39.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:39.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:39.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:39.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:39.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:39.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:39.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:39.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:40.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:40.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:40.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:40.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:40.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:40.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:40.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:40.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:40.651+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:40.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:40.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:40.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:40.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:41.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:41.001+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31392 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:41.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:41.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:41.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:41.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:41.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:41.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:41.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:41.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:41.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:41.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:41.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:41.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:41.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:41.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:41.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:41.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:42.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:42.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:42.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:42.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:42.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:42.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:42.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:42.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:42.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:42.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:42.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:42.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:42.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:42.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:43.570 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:43.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:43.571 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:43.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:43.571 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:43.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:43.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:43.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:43.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:43.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:43.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:43.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:43.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:44.662 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:44.253+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:44.663 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:44.253+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:44.663 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:44.253+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:44.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:44.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:44.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:44.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:44.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:44.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:44.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:44.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:45.635 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:45.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:45.635 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:45.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:45.635 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:45.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:45.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:45.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:45.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:45.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:45.711+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:45.711+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:45.711+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:46.261 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:46.006+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31465 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:46.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:46.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:46.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:46.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:46.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:46.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:46.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:46.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:46.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:46.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:46.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:46.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:46.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:46.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:46.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:46.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:47.573 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:47.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:47.573 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:47.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:47.573 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:47.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:47.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:47.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:47.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:47.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:47.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:47.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:47.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:47.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:47.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:47.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:47.572+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:48.612 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:48.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:48.613 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:48.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:48.613 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:48.270+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:48.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:48.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:48.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:48.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:48.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:48.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:48.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:48.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:48.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:48.612+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:49.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:49.264+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:49.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:49.264+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:49.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:49.264+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:49.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:49.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:49.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:49.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:49.642+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:49.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:49.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:49.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:49.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:50.531 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:50.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:50.531 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:50.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:50.531 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:50.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:50.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:50.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:50.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:50.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:50.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:50.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:50.531+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:50.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:50.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:50.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:51.267 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:51.011+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31539 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:51.572 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:51.267+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:51.572 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:51.267+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:51.572 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:51.267+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:51.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:51.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:51.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:51.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:51.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:51.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:51.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:51.572+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:51.828 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:51.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:51.828 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:51.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:51.828 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:51.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:52.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:52.540+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:52.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:52.540+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:52.633 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:52.540+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:52.633 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:52.540+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:52.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:52.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:52.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:52.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:52.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:52.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:52.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:52.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:52.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:53.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:53.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:53.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:53.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:53.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:53.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:53.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:53.533+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:53.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:53.533+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:53.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:53.533+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:53.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:53.533+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:53.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:53.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:53.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:54.613 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:54.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:54.614 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:54.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:54.614 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:54.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:54.614 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:54.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:54.614 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:54.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:54.614 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:54.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:54.614 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:54.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:54.899 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:54.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:54.899 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:54.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:54.899 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:54.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:55.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:55.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:55.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:55.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:55.588 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:55.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:55.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:55.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:55.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:55.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:55.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:55.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:55.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:55.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:55.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:55.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:56.271 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:50:56.017+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31603 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:50:56.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:56.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:56.561 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:56.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:56.561 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:56.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:56.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:56.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:56.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:56.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:56.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:56.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:56.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:56.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:56.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:56.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:57.549 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:57.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:57.549 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:57.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:57.549 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:57.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:57.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:57.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:57.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:57.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:57.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:57.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:57.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:58.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:58.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:58.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:58.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:58.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:58.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:58.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:58.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:58.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:58.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:58.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:58.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:58.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:59.576 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:59.254+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:59.576 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:59.254+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:59.576 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:50:59.254+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:50:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:59.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:59.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:59.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:50:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:50:59.575+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:59.732+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:59.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:50:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:50:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:50:59.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:00.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:00.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:00.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:00.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:00.600 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:00.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:00.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:00.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:00.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:00.600+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:00.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:00.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:00.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:01.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:01.022+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31677 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:01.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:01.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:01.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:01.215+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:01.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:01.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:01.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:01.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:01.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:01.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:01.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:02.591 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:02.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:02.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:02.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:02.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:02.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:02.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:02.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:02.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:02.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:02.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:02.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:02.727+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:03.582 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:03.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:03.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:03.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:03.583 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:03.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:03.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:03.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:03.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:03.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:03.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:03.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:03.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:03.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:04.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:04.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:04.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:04.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:04.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:04.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:04.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:04.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:04.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:04.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:04.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:04.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:04.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:04.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:05.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:05.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:05.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:05.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:05.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:05.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:05.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:05.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:05.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:05.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:05.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:05.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:05.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:05.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:06.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:06.027+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31749 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:06.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:06.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:06.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:06.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:06.756 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:06.757 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:06.757 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:06.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:06.757 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:06.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:07.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:06.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:07.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:06.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:07.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:06.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:07.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:07.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:07.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:07.192+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:07.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:07.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:07.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:07.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:07.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:07.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:07.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:07.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:07.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:07.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:08.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:08.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:08.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:08.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:08.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:08.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:08.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:08.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:08.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:08.556+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:08.556+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:09.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:09.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:09.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:09.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:09.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:09.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:09.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:09.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:09.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:09.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:09.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:10.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:10.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:10.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:10.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:10.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:10.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:10.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:10.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:10.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:10.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:10.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:10.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:10.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:10.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:11.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:11.032+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31823 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:11.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:11.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:11.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:11.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:11.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:11.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:11.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:11.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:11.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:11.665+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:12.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:12.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:12.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:12.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:12.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:12.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:12.738+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:12.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:12.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:12.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:12.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:13.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:13.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:13.542 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:13.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:13.542 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:13.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:13.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:13.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:13.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:13.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:13.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:13.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:13.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:14.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:14.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:14.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:14.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:15.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:14.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:15.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:14.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:15.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:14.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:15.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:14.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:15.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:14.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:15.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:14.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:15.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:14.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:15.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:15.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:15.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:15.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:15.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:15.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:15.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:15.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:15.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:15.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:16.403 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:16.036+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31895 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:16.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:16.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:16.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:16.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:16.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:16.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:16.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:16.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:16.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:16.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:16.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:16.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:16.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:17.150+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:17.150+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:17.150+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:17.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:17.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:17.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:17.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:17.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:17.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:17.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:18.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:18.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:18.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:18.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:18.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:18.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:18.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:18.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:18.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:18.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:19.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:19.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:19.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:19.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:19.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:19.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:19.748+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:19.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:19.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:19.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:19.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:20.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:20.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:20.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:21.044 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:20.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:21.045 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:20.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:21.045 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:20.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:21.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:20.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:21.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:20.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:21.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:20.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:21.046 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:20.736+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:21.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:21.041+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 31969 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:21.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:21.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:21.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:21.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:21.101+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:22.106 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:21.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:22.106 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:21.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:22.106 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:21.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:22.106 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:21.776+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:22.106 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:21.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:22.106 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:21.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:22.107 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:21.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:22.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:22.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:22.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:22.105+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:22.810 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:22.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:22.810 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:22.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:22.810 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:22.733+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:23.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:22.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:23.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:22.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:23.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:22.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:23.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:22.809+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:23.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:23.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:23.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:23.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:23.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:23.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:23.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:23.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:23.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:23.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:23.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:24.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:24.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:24.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:24.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:24.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:24.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:24.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:24.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:24.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:24.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:24.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:24.794+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:24.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:25.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:25.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:25.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:25.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:25.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:25.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:25.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:25.708+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:25.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:25.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:25.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:25.772+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:26.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:26.046+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32042 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:26.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:26.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:26.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:26.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:26.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:26.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:26.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:26.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:26.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:26.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:26.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:27.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:27.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:27.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:28.073 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:27.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:28.073 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:27.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:28.073 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:27.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:27.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:27.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:27.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:27.758+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:28.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:28.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:28.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:28.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:28.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:28.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:28.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:28.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:28.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:28.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:29.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:29.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:29.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:29.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:29.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:29.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:29.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:29.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:29.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:29.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:29.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:29.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:29.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:30.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:30.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:30.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:30.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:30.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:30.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:30.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:30.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:30.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:30.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:30.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:30.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:30.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:31.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:31.051+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32115 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:31.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:31.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:31.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:31.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:31.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:32.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:31.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:32.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:31.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:32.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:31.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:32.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:31.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:32.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:31.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:32.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:31.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:32.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:31.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:32.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:32.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:32.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:33.016 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:32.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:33.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:32.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:33.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:32.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:33.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:32.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:33.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:32.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:33.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:32.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:33.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:32.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:33.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:33.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:33.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:33.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:33.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:33.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:33.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:33.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:33.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:33.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:33.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:33.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:33.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:33.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:33.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:33.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:33.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:33.995 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:33.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:34.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:34.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:34.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:34.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:34.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:34.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:34.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:35.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:34.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:34.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:35.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:34.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:35.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:34.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:35.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:34.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:34.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:34.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:34.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:34.832+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:36.129 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:36.056+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32188 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:36.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:35.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:36.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:35.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:36.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:35.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:36.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:35.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:36.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:35.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:36.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:35.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:36.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:35.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:36.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:35.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:36.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:35.872+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:36.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:35.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:36.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:37.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:36.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:37.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:36.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:36.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:36.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:36.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:36.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:36.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:36.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:36.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:37.914+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:37.914+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:38.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:37.914+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:38.231 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:37.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:38.231 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:37.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:38.231 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:37.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:38.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:37.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:38.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:37.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:38.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:37.907+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:38.232 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:37.907+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:38.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:39.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:38.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:38.878+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:38.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:38.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:38.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:38.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:38.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:38.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:38.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:40.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:39.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:39.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:39.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:39.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:39.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:39.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:40.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:39.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:40.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:39.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:40.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:39.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:40.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:39.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:41.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:41.062+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32260 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:41.074 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:40.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:41.074 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:40.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:41.074 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:40.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:41.074 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:40.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:41.074 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:40.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:41.075 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:40.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:41.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:40.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:41.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:40.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:41.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:40.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:41.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:40.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:42.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:41.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:42.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:41.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:42.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:41.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:41.843+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:41.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:41.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:41.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:42.006+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:42.006+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:42.006+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:43.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:43.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:43.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:43.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:42.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:42.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:42.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:43.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:42.841+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:42.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:42.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:42.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:43.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:43.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:43.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:43.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:43.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:43.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:43.795+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:44.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:44.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:44.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:44.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:45.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:44.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:45.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:44.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:45.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:44.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:45.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:44.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:45.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:44.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:45.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:44.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:45.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:44.773+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:45.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:45.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:45.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:45.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:45.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:45.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:46.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:45.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:46.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:45.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:46.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:45.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:46.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:45.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:46.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:45.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:46.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:45.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:46.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:45.787+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:46.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:46.068+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32333 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:46.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:46.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:46.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:46.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:47.018 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:46.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:47.018 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:46.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:47.018 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:46.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:47.018 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:46.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:47.018 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:46.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:47.018 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:46.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:47.019 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:46.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:47.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:47.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:47.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:47.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:47.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:47.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:48.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:47.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:48.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:47.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:48.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:47.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:48.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:47.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:48.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:47.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:48.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:47.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:48.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:47.779+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:48.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:48.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:48.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:49.014 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:48.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:49.014 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:48.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:49.014 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:48.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:49.014 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:48.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:49.014 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:48.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:49.014 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:48.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:49.015 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:48.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:49.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:49.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:49.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:49.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:50.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:49.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:50.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:49.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:50.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:49.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:50.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:49.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:50.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:49.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:50.048 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:49.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:50.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:49.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:50.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:50.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:50.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:50.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:50.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:50.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:50.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:50.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:50.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:50.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:50.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:50.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:50.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:50.834+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:51.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:51.074+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32406 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:51.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:51.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:51.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:51.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:51.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:51.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:51.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:51.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:51.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:51.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:51.679+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:52.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:52.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:52.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:52.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:52.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:52.675+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:52.675+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:52.675+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:52.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:52.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:52.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:52.900+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:53.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:53.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:53.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:53.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:53.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:53.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:53.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:53.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:53.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:53.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:53.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:53.903+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:53.903+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:54.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:54.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:54.494 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:54.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:54.494 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:54.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:54.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:54.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:54.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:54.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:54.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:54.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:55.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:55.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:55.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:55.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:54.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:54.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:54.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:54.950+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:55.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:55.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:55.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:55.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:55.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:55.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:56.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:51:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:51:56.079+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32479 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:51:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:55.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:55.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:55.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:55.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:55.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:55.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:55.974+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:56.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:56.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:56.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:57.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:57.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:57.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:57.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:57.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:57.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:57.019+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:57.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:57.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:57.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:57.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:57.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:57.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:57.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:57.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:57.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.020+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:58.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:58.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:58.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:58.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:59.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:59.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:59.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:59.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:51:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:51:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:51:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:51:58.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:59.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:59.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:59.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:59.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:51:59.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:51:59.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:00.271 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:59.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:00.272 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:59.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:00.272 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:51:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:51:59.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:00.272 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:00.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:00.272 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:00.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:00.272 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:00.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:00.272 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:00.031+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:00.975 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:00.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:00.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:00.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:00.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:00.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:01.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:01.084+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32552 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:00.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:00.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:00.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:01.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:01.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:01.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:01.029+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:01.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:01.625+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:01.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:01.625+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:01.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:01.625+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:01.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:02.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:01.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:01.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:02.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:02.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:02.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:02.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:02.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:02.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:02.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:02.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:02.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:02.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:02.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:02.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:02.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:03.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:02.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:03.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:02.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:03.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:02.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:03.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:03.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:03.078+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:03.078+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:03.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:03.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:03.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:03.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:03.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:03.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:04.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:03.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:03.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:03.957+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:04.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:04.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:04.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:04.072+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:04.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:04.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:04.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:04.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:04.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:05.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:04.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:04.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:05.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:05.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:05.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:05.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:05.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:05.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:05.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:05.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:05.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:05.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:06.213 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:06.088+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32625 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:06.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:05.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:06.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:05.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:06.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:05.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:06.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:06.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:06.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:06.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:06.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:06.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:06.214 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:06.037+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:06.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:06.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:06.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:06.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:06.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:06.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:07.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:07.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.018+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:06.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:06.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:06.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:07.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:07.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:07.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:07.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:07.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:07.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:08.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:07.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:08.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:07.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:07.970+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.991+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:07.991+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:08.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:08.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:08.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:08.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:09.283 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:09.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:09.283 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:09.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:09.283 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:09.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:09.283 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:09.031+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:09.284 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:08.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:09.284 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:08.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:09.284 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:08.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:09.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:09.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:09.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:09.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:10.272 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:09.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:10.272 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:09.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:10.272 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:09.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:10.272 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:10.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:10.272 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:10.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:10.273 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:10.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:10.273 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:10.055+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:10.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:10.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:10.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:10.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:10.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:10.638+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:11.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:11.096+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32698 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:11.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:11.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:11.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:11.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:11.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:11.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:11.029+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:11.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:11.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:11.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:12.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:12.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:12.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:12.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:12.022+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:12.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:12.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:12.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:12.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:12.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:12.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:12.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:13.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:13.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:13.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:13.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:13.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:13.056+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:13.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:13.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:13.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:13.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:13.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:13.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:13.658+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:14.325 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:14.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:14.325 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:14.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:14.325 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:14.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:14.325 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:14.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:14.326 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:14.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:14.326 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:14.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:14.326 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:14.094+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:14.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:14.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:14.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:14.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:14.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:14.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:15.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:15.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:15.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:15.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:15.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:15.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:15.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:15.073+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:15.639+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:15.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:15.639+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:15.639+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:16.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:16.102+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32771 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:16.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:16.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:16.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:16.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:16.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:16.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:16.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:16.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:16.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:16.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:17.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:17.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:17.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:17.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:17.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:17.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:17.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:17.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:17.039+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:17.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:17.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:17.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:18.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:18.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:18.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:18.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:18.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:18.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:18.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:18.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:18.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:18.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:18.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:18.060+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:18.607+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:18.607+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:18.607+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:19.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:19.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:19.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:19.180+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:19.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:19.180+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:19.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:19.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:19.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:19.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:19.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:19.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:19.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:19.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:19.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:19.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:19.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:19.582+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:20.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:20.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:20.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:20.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:20.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:20.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:20.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:20.026+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:20.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:20.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:20.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:20.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:20.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:20.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:21.208 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:21.106+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32844 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:21.208 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:21.068+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:21.209 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:21.068+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:21.209 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:21.068+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:21.209 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:21.068+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:21.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:21.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:21.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:21.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:21.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:21.630+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:21.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:21.630+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:21.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:21.630+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:22.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:22.033+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:22.033+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:22.033+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:22.033+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:22.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:22.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:22.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:22.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:22.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:22.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:22.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:23.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:23.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:23.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:23.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:23.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:23.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:23.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:23.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:23.604+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:23.604+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:23.604+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:24.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:24.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:24.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:24.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:24.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:24.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:24.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:24.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:24.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:24.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:24.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:24.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:24.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:24.067+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:24.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:24.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:24.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:24.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:24.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:24.608+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:25.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:25.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:25.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:25.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:25.029+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:25.233+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:25.233+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:25.233+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:25.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:25.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:25.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:25.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:26.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:26.112+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32917 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:26.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:26.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:26.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:26.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:26.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:26.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:26.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:26.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:26.020+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:26.020+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:26.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:26.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:26.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:26.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:27.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:27.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:27.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:27.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:27.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:27.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:27.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:27.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:27.065+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:27.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:27.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:27.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:27.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:28.147+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:28.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:28.147+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:28.147+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:28.097+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:28.097+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:28.097+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:28.097+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:28.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:28.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:28.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:28.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:29.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:29.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:29.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:29.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:29.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:29.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:29.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:29.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:29.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:29.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:29.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:29.057+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:29.452 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:29.058+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:29.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:29.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:29.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:29.672+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:30.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:30.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:30.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:30.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:30.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:30.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:30.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:30.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:30.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:30.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:30.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:30.098+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:30.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:30.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:30.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:30.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:30.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:30.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:31.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:31.118+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 32990 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:31.159+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:31.159+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:31.159+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:31.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:31.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:31.114+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:31.114+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:31.626+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:31.626+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:31.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:31.626+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:32.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:32.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:32.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:32.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:32.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:32.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:32.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:32.162+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:32.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:32.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:32.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:32.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:32.612+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:33.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:33.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:33.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:33.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:33.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:33.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:33.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:33.188+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:33.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:33.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:33.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:33.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:33.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:33.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:34.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:34.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:34.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:34.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:34.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:34.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:34.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:34.178+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:34.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:34.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:34.097+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:34.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:34.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:34.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:34.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:35.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:35.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:35.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:35.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:35.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:35.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:35.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:35.169+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:35.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:35.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:35.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:35.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:35.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:35.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:35.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:36.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:36.123+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33063 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:36.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:36.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:36.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:36.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:36.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:36.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:36.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:36.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:36.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:36.620+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:37.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:37.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:37.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:37.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:37.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:37.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:37.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:37.192+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:37.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:37.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:37.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:37.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:37.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:37.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:37.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:37.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:37.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:37.628+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:38.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:38.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:38.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:38.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:38.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:38.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:38.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:38.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:38.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:38.184+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:38.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:38.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:38.581+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:39.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:39.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:39.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:39.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:39.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:39.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:39.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:39.161+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:39.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:39.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:39.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:39.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:39.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:40.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:40.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:40.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:40.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:40.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:40.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:40.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:40.177+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:40.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:40.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:40.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:40.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:40.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:40.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:41.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:41.128+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33135 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:41.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:41.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:41.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:41.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:41.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:41.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:41.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:41.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:41.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:41.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:41.188+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:41.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:41.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:41.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:41.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:42.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:42.203+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:42.203+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:42.203+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:42.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:42.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:42.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:42.233+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:42.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:42.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:42.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:43.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:43.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:43.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:43.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:43.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:43.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:43.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:43.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:43.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:43.256+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:43.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:43.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:43.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:43.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:44.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:44.225+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:44.225+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:44.225+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:44.260+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:44.260+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:44.260+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:44.260+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:44.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:44.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:44.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:44.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:45.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:45.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:45.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:45.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:45.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:45.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:45.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:45.264+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:45.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:45.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:45.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:45.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:45.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:46.450 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:46.133+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33209 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:46.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:46.200+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:46.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:46.200+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:46.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:46.200+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:46.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:46.236+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:46.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:46.236+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:46.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:46.236+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:46.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:46.236+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:46.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:46.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:46.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:47.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:47.152+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:47.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:47.152+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:47.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:47.152+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:47.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:47.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:47.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:47.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:47.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:47.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:47.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:47.233+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:47.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:47.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:47.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:47.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:47.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:48.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:48.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:48.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:48.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:48.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:48.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:48.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:48.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:48.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:48.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:48.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:48.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:48.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:48.268+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:48.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:48.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:48.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:48.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:49.618 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:49.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:49.619 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:49.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:49.619 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:49.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:49.619 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:49.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:49.619 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:49.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:49.619 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:49.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:49.619 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:49.263+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:49.620 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:49.263+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:49.620 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:49.263+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:49.620 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:49.263+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:50.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:50.287+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:50.287+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:50.287+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:50.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:50.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:50.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:50.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:50.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:50.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:50.243+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:51.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:51.138+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33281 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:51.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:51.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:51.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:51.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:51.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:51.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:51.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:51.258+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:51.841 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:51.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:51.841 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:51.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:51.841 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:51.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:52.536 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:52.248+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:52.536 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:52.248+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:52.537 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:52.248+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:52.537 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:52.248+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:52.537 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:52.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:52.537 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:52.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:52.537 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:52.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:52.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:52.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:52.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:53.711 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:53.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:53.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:53.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:53.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:53.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:53.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:53.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:53.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:53.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:53.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:53.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:53.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:53.295+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:53.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:53.295+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:53.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:53.295+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:53.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:53.296+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:54.588 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:54.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:54.589 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:54.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:54.589 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:54.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:54.589 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:54.270+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:54.589 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:54.270+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:54.589 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:54.270+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:54.589 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:54.270+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:54.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:54.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:54.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:54.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:54.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:55.547 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:55.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:55.547 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:55.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:55.548 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:55.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:55.548 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:55.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:55.548 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:55.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:55.548 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:55.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:55.548 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:55.293+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:55.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:55.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:55.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:55.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:55.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:55.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:56.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:52:56.143+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33355 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:52:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:56.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:56.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:56.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:56.257+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:56.257+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:56.257+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:56.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:56.257+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:56.594+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:56.594+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:56.594+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:57.614 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:57.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:57.614 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:57.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:57.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:57.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:57.615 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:57.289+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:57.615 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:57.289+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:57.615 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:57.289+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:57.615 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:57.289+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:57.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:57.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:57.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:57.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:57.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:58.628 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:58.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:58.628 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:58.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:58.628 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:58.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:58.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:58.323+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:58.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:58.323+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:58.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:58.323+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:58.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:58.323+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:58.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:58.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:58.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:58.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:59.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:59.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:59.588 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:59.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:59.588 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:52:59.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:52:59.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:59.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:59.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:59.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:59.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:59.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:52:59.589 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:52:59.321+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:59.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:59.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:59.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:59.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:52:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:52:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:52:59.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:00.601 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:00.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:00.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:00.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:00.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:00.340+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:00.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:00.313+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:00.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:00.313+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:00.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:00.313+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:00.602 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:00.313+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:00.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:00.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:00.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:00.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:00.852 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:00.600+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:01.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:01.149+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33427 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:01.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:01.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:01.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:01.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:01.360+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:01.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:01.360+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:01.360+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:01.360+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:01.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:01.578+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:01.578+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:01.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:01.578+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:02.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:02.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:02.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:02.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:02.588 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:02.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:02.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:02.345+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:02.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:02.345+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:02.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:02.345+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:02.588 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:02.345+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:02.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:02.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:02.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:02.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:02.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:02.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:03.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:03.555+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:03.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:03.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:03.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:03.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:03.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:03.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:03.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:03.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:03.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:03.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:03.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:03.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:03.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:03.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:03.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:03.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:03.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:03.378+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:04.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:04.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:04.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:04.351+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:04.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:04.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:04.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:04.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:04.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:04.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:05.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:05.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:05.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:05.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:05.325+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:05.325+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:05.325+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:05.325+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:05.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:05.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:05.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:06.436 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:06.153+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33501 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:06.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:06.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:06.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:06.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:06.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:06.371+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:06.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:06.371+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:06.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:06.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:06.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:06.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:06.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:06.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:06.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:07.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:07.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:07.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:07.403+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:07.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:07.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:07.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:07.341+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:07.341+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:07.341+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:07.341+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:08.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:08.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:08.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:08.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:08.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:08.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:08.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:08.355+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:08.355+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:08.355+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:08.355+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:09.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:09.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:09.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:09.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:09.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:09.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:09.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:09.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:09.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:09.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:09.340+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:10.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:10.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:10.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:10.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:10.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:10.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:10.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:10.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:10.356+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:10.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:10.356+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:10.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:10.356+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:10.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:10.356+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:11.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:11.158+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33573 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:11.314+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:11.314+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:11.314+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:11.314+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:11.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:11.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:11.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:11.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:11.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:11.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:11.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:12.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:12.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:12.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:12.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:12.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:12.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:12.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:12.308+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:12.308+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:12.308+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:12.308+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:13.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:13.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:13.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:13.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:13.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:13.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:13.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:13.261+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:13.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:13.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:13.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:13.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:13.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:13.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:13.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:13.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:13.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:13.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:13.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:13.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:14.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:14.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:14.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:14.304+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:14.304+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:14.304+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:14.304+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:14.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:14.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:14.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:15.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:15.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:15.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:15.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:15.346+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:15.346+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:15.346+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:15.346+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:15.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:15.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:15.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:16.476 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:16.163+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33647 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:16.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:16.390+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:16.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:16.390+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:16.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:16.390+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:16.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:16.390+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:16.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:16.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:16.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:16.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:16.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:16.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:16.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:16.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:16.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:16.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:17.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:17.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:17.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:17.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:17.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:17.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:17.538+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:17.407+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:17.407+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:17.407+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:17.407+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:18.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:18.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:18.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:18.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:18.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:18.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:18.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:18.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:18.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:18.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:18.444+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:19.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:19.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:19.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:19.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:19.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:19.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:19.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:19.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:19.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:19.492+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:20.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:20.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:20.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:20.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:20.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:20.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:20.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:20.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:20.477+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:20.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:20.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:20.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:21.448 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:21.169+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33719 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:21.448 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:21.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:21.448 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:21.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:21.449 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:21.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:21.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:21.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:21.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:21.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:21.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:21.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:21.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:21.447+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:22.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:22.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:22.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:22.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:22.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:22.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:22.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:22.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:22.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:22.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:22.478+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:23.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:23.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:23.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:23.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:23.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:23.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:23.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:23.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:23.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:23.501+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:23.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:23.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:23.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:23.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:23.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:23.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:24.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:24.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:24.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:24.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:24.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:24.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:24.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:24.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:24.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:24.543+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:24.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:24.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:24.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:25.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:25.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:25.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:25.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:25.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:25.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:25.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:25.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:25.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:25.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:25.520+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:26.448 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:26.174+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33793 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:26.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:26.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:26.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:26.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:26.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:26.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:26.464+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:26.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:26.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:26.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:26.484+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:27.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:27.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:27.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:27.494+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:27.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:27.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:27.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:27.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:27.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:27.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:27.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:28.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:28.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:28.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:28.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:28.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:28.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:28.487+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:28.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:28.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:28.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:28.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:28.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:28.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:29.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:29.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:29.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:29.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:29.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:29.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:29.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:29.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:29.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:29.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:29.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:29.559+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:30.542 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:30.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:30.542 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:30.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:30.542 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:30.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:30.542 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:30.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:30.542 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:30.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:30.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:30.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:30.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:30.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:30.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:30.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:30.541+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:31.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:31.180+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33865 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:31.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:31.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:31.405+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:31.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:31.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:31.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:31.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:31.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:31.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:31.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:31.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:31.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:31.501+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:32.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:32.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:32.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:32.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:32.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:32.563+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:32.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:32.563+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:32.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:32.563+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:32.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:32.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:32.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:32.530+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:33.573 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:33.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:33.574 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:33.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:33.574 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:33.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:33.574 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:33.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:33.574 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:33.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:33.574 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:33.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:33.574 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:33.569+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:33.573+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:33.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:33.573+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:33.573+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:34.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:34.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:34.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:34.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:34.581 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:34.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:34.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:34.579+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:34.579+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:34.579+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:34.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:34.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:34.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:34.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:35.539 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:35.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:35.540 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:35.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:35.540 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:35.282+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:35.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:35.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:35.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:35.539+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:35.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:35.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:35.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:35.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:36.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:36.185+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 33939 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:36.293+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:36.293+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:36.293+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:36.504+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:36.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:36.504+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:36.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:36.504+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:36.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:36.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:36.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:36.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:37.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:37.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:37.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:37.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:37.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:37.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:37.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:37.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:37.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:37.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:37.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:37.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:37.504+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:38.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:38.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:38.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:38.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:38.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:38.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:38.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:38.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:38.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:38.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:38.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:38.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:39.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:39.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:39.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:39.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:39.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:39.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:39.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:39.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:39.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:39.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:39.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:39.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:39.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:39.467+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:40.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:40.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:40.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:40.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:40.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:40.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:40.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:40.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:40.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:40.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:40.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:41.470 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:41.190+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34011 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:41.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:41.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:41.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:41.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:41.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:41.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:41.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:41.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:41.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:41.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:41.469+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:41.469+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:41.469+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:41.469+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:42.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:42.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:42.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:42.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:42.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:42.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:42.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:42.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:42.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:42.464+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:43.715 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:43.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:43.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:43.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:43.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:43.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:43.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:43.451+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:43.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:43.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:43.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:43.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:43.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:43.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:43.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:43.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:43.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:43.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:43.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:43.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:44.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:44.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:44.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:44.317+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:44.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:44.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:44.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:44.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:44.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:44.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:44.425+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:45.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:45.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:45.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:45.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:45.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:45.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:45.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:45.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:45.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:45.475+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:45.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:45.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:45.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:45.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:45.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:45.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:46.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:46.195+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34086 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:46.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:46.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:46.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:46.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:46.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:46.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:46.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:46.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:46.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:46.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:46.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:46.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:46.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:46.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:46.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:46.506+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:47.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:47.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:47.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:47.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:47.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:47.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:47.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:47.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:47.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:47.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:47.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:47.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:47.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:47.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:47.505+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:48.555 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:48.303+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:48.556 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:48.303+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:48.556 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:48.303+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:48.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:48.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:48.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:48.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:48.557 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:48.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:48.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:48.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:48.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:48.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:48.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:49.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:49.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:49.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:49.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:49.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:49.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:49.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:49.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:49.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:49.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:49.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:50.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:50.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:50.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:50.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:50.436 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:50.306+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:50.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:50.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:50.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:50.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:50.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:50.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:50.539+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:50.539+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:51.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:51.200+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34158 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:51.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:51.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:51.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:51.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:51.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:51.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:51.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:51.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:51.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:51.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:52.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:52.298+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:52.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:52.298+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:52.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:52.298+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:52.599 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:52.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:52.599 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:52.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:52.599 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:52.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:52.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:52.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:52.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:52.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:52.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:53.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:53.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:53.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:53.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:53.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:53.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:53.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:53.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:53.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:53.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:53.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:53.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:53.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:53.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:53.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:53.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:53.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:53.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:53.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:53.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:54.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:54.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:54.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:54.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:54.521+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:54.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:54.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:54.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:54.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:54.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:54.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:54.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:55.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:55.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:55.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:55.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:55.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:55.269+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:55.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:55.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:55.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:55.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:55.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:55.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:55.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:56.424 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:53:56.206+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34232 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:53:56.425 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:56.247+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:56.425 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:56.247+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:56.425 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:56.247+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:56.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:56.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:56.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:56.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:56.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:56.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:56.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:56.551+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:57.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:57.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:57.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:57.224+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:57.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:57.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:57.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:57.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:57.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:57.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:57.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:57.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:58.628 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:58.247+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:58.628 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:58.247+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:58.628 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:58.247+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:58.629 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:58.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:58.629 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:58.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:58.629 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:58.418+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:58.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:58.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:58.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:58.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:58.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:59.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:59.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:59.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:59.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:59.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:53:59.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:53:59.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:59.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:59.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:59.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:59.616 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:53:59.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:53:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:59.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:59.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:59.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:53:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:53:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:53:59.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:00.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:00.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:00.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:00.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:00.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:00.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:00.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:00.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:00.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:00.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:00.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:00.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:00.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:01.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:01.211+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34304 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:01.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:01.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:01.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:01.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:01.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:01.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:01.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:01.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:01.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:01.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:01.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:01.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:02.225+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:02.225+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:02.225+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:02.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:02.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:02.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:02.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:02.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:02.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:02.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:02.510+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:03.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:03.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:03.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:03.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:03.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:03.246+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:03.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:03.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:03.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:03.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:03.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:03.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:03.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:03.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:03.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:03.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:03.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:03.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:03.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:03.489+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:04.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:04.279+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:04.279+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:04.279+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:04.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:04.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:04.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:04.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:04.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:04.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:04.443+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:05.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:05.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:05.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:05.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:05.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:05.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:05.393+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:05.394+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:05.394+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:05.394+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:06.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:06.216+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34378 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:06.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:06.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:06.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:06.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:06.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:06.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:06.347+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:06.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:06.347+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:06.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:06.347+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:06.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:06.347+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:07.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:07.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:07.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:07.323+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:07.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:07.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:07.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:07.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:07.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:07.372+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:07.372+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:08.279+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:08.279+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:08.279+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:08.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:08.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:08.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:08.347+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:08.347+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:08.347+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:08.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:08.348+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:09.258+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:09.259+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:09.259+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:09.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:09.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:09.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:09.339+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:09.339+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:09.339+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:09.340+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:10.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:10.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:10.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:10.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:10.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:10.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:10.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:10.355+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:10.355+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:10.355+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:10.355+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:11.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:11.222+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34450 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:11.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:11.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:11.265+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:11.343+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:11.343+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:11.343+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:11.343+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:11.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:11.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:11.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:11.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:11.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:12.332+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:12.332+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:12.332+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:12.332+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:12.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:12.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:12.299+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:12.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:12.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:12.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:13.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:13.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:13.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:13.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:13.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:13.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:13.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:13.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:13.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:13.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:13.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:13.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:13.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:13.337+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:13.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:13.337+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:13.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:13.337+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:13.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:13.337+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:14.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:14.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:14.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:14.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:14.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:14.281+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:14.580 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:14.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:14.581 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:14.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:14.581 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:14.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:14.581 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:14.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:14.581 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:14.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:14.581 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:14.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:14.581 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:14.387+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:15.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:15.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:15.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:15.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:15.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:15.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:15.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.944169+0000 (oldest deadline 2024-02-19T19:44:00.944169+0000) 2024-02-19T19:54:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:15.357+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:15.357+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:15.357+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:15.357+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:16.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:16.228+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34524 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:16.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:16.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:16.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:40.289042+0000 (oldest deadline 2024-02-19T19:44:00.289042+0000) 2024-02-19T19:54:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:16.365+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:16.365+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:16.365+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:16.366+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:17.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:17.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:17.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:17.408+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:17.408+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:18.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:18.427+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:18.427+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:18.427+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:18.427+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:19.402+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:19.402+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:19.402+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:19.402+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:20.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:20.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:20.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:20.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:20.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:20.423+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:20.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:20.423+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:21.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:21.233+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34596 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:21.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:21.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:21.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:21.449+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:22.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:22.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:22.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:22.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:22.404+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:23.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:23.429+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:23.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:23.429+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:23.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:23.429+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:23.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:23.430+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:24.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:24.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:24.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:24.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:24.406+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:25.376+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:25.376+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:25.376+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:25.376+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:26.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:26.238+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34670 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:26.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:26.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:26.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:26.365+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:27.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:27.319+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:27.319+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:27.319+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:27.319+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:28.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:28.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:28.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:28.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:28.340+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:29.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:29.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:29.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:29.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:43:53.834832+0000 (oldest deadline 2024-02-19T19:44:13.834832+0000) 2024-02-19T19:54:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:29.294+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:30.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:30.312+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:31.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:31.242+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34742 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:31.362+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:32.315+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:33.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:33.302+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:34.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:34.277+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:35.318+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:36.371 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:36.247+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34816 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:36.371 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:36.285+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:37.256+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:38.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:38.276+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:39.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:39.276+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:40.541 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:40.272+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:41.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:41.253+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34888 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:41.316+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:41.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:41.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:41.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:41.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:41.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:41.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:42.529 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:42.310+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:42.530 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:42.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:42.530 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:42.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:42.530 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:42.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:42.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:42.528+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:42.528+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:42.528+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:43.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:43.279+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:43.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:43.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:43.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:43.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:43.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:43.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:43.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:43.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:43.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:44.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:44.326+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:44.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:44.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:44.533+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:44.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:44.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:44.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:45.569 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:45.288+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:45.570 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:45.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:45.570 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:45.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:45.570 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:45.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:45.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:45.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:45.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:45.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:46.612 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:46.258+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 34962 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:46.612 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:46.259+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:46.612 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:46.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:46.613 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:46.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:46.613 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:46.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:46.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:46.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:46.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:46.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:47.633 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:47.282+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:47.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:47.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:47.633 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:47.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:47.634 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:47.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:47.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:47.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:47.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:47.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:47.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:47.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:48.643 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:48.317+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:48.643 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:48.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:48.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:48.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:48.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:48.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:48.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:48.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:48.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:48.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:48.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:48.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:49.638 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:49.338+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:49.639 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:49.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:49.639 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:49.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:49.639 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:49.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:49.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:49.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:49.637+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:50.625 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:50.370+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:50.625 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:50.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:50.625 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:50.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:50.626 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:50.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:50.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:50.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:50.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:50.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:51.702 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:51.263+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35034 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:51.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:51.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:51.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:51.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:51.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:51.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:51.703 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:51.340+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:51.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:51.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:51.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:51.701+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:52.674 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:52.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:52.674 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:52.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:52.674 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:52.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:52.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:52.298+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:52.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:52.298+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:52.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:52.298+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:52.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:52.298+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:52.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:52.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:52.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:52.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:53.649 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:53.283+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:53.649 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:53.283+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:53.649 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:53.283+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:53.649 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:53.283+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:53.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:53.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:53.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:53.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:53.650 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:53.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:53.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:53.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:53.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:53.647+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:54.516 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:54.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:54.516 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:54.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:54.516 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:54.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:54.516 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:54.254+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:54.517 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:54.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:54.517 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:54.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:54.517 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:54.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:54.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:54.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:54.688+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:55.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:55.215+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:55.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:55.215+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:55.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:55.215+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:55.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:55.215+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:55.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:55.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:55.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:55.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:55.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:55.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:55.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:55.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:55.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:56.477 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:54:56.270+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35108 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:54:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:56.216+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:56.216+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:56.216+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:56.216+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:56.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:56.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:56.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:56.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:56.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:56.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:56.681+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:57.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:57.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:57.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:57.224+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:57.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:57.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:57.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:57.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:57.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:57.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:58.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:58.237+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:58.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:58.237+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:58.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:58.237+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:58.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:58.238+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:58.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:58.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:58.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:58.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:58.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:58.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:58.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:58.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:58.723+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:54:59.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:59.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:59.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:59.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:59.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:59.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:54:59.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:54:59.274+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:59.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:59.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:54:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:54:59.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:00.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:59.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:59.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:54:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:54:59.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:00.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:00.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:00.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:00.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:00.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:00.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:00.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:00.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:00.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:00.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:00.660 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:00.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:00.661 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:00.233+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:00.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:00.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:00.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:00.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:00.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:01.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:01.275+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35180 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:01.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:01.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:01.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:01.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:01.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:01.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:01.277+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:01.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:01.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:01.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:02.284+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:02.284+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:02.284+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:02.284+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:02.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:02.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:02.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:03.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:02.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:02.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:02.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:03.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:03.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:03.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:03.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:03.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:03.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:03.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:03.251+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:03.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:03.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:03.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:03.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:03.721 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:03.304+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:03.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:03.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:03.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:03.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:04.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:04.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:04.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:04.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:04.233+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:04.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:04.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:04.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:04.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:04.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:04.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:05.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:05.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:05.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:05.251+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:05.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:05.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:05.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:06.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:05.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:05.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:05.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:06.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:06.279+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35254 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:06.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:06.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:06.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:06.282+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:06.282+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:06.282+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:06.282+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:07.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:06.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:06.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:06.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:07.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:07.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:07.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:07.306+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:07.306+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:07.306+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:07.307+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:07.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:07.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:07.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:08.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:08.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:08.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:08.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:08.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:08.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:08.322+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:08.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:08.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:08.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:09.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:09.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:09.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:09.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:09.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:09.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:09.274+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:10.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:09.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:10.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:09.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:10.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:09.734+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:10.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:10.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:10.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:10.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:10.279+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:10.279+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:10.279+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:10.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:10.279+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:10.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:10.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:10.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:11.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:11.284+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35326 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:11.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:11.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:11.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:11.246+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:11.246+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:11.246+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:11.246+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:12.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:11.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:12.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:11.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:12.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:11.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:12.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:12.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:12.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:12.222+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:12.222+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:12.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:12.222+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:12.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:12.222+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:13.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:12.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:13.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:12.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:13.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:12.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:13.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:13.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:13.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:13.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:13.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:13.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:13.224+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:13.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:13.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:13.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:14.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:14.417+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:14.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:14.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:14.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:14.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:14.254+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:15.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:14.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:14.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:14.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:15.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:15.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:15.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:15.248+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:15.248+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:15.248+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:15.248+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:16.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:15.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:16.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:15.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:16.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:15.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:16.717 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:16.290+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35400 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:16.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:16.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:16.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:16.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:16.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:16.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:16.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:16.284+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:16.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:16.284+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:16.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:16.284+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:16.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:16.284+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:16.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:16.717+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:16.717+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:16.717+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:17.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:17.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:17.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:17.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:17.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:17.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:17.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:17.269+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:17.721 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:17.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:17.721 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:17.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:17.721 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:17.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:17.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:17.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:17.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:17.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:18.515 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:18.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:18.515 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:18.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:18.515 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:18.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:18.516 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:18.250+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:18.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:18.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:18.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:18.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:18.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:18.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:19.492 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:19.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:19.492 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:19.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:19.492 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:19.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:19.492 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:19.235+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:19.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:19.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:19.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:19.718+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:19.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:19.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:19.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:20.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:20.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:20.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:20.195+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:20.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:20.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:20.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:20.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:20.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:20.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:20.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:20.691+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:20.878 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:20.692+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:20.878 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:20.692+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:21.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:21.294+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35472 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:21.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:21.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:21.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:21.200+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:21.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:21.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:21.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:21.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:21.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:21.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:21.678+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:22.229+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:22.229+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:22.229+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:22.229+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:22.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:22.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:22.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:22.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:22.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:22.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:23.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:23.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:23.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:23.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:23.268+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:23.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:23.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:23.714 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:23.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:23.715 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:23.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:23.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:23.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:23.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:23.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:24.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:24.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:24.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:24.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:24.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:24.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:24.256+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:25.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:24.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:24.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:24.753+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:25.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:25.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:25.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:25.292+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:25.292+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:25.292+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:25.292+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:25.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:25.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:25.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:26.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:26.300+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35546 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:26.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:26.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:26.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:26.318+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:26.318+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:26.318+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:26.318+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:26.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:26.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:26.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:27.354+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:27.354+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:27.354+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:27.354+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:27.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:27.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:27.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:28.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:28.324+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:28.324+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:28.324+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:28.324+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:28.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:28.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:28.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:28.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:28.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:28.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:29.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:29.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:29.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:29.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:29.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:29.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:29.364+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:29.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:29.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:29.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:30.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:30.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:30.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:30.412+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:30.412+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:30.412+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:30.412+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:31.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:30.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:30.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:30.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:31.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:31.306+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35618 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:31.377+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:31.377+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:31.377+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:31.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:31.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:31.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:31.454+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:32.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:31.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:31.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:31.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:32.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:32.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:32.332+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:32.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:32.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:32.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:32.447+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:33.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:32.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:32.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:32.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:33.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:33.438+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:33.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:33.438+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:33.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:33.438+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:33.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:33.438+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:33.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:33.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:33.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:33.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:33.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:33.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:33.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:33.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:33.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:34.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:34.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:34.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:34.459+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:34.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:34.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:34.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:34.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:34.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:34.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:35.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:35.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:35.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:35.433+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:35.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:35.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:35.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:35.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:35.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:35.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:36.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:36.311+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35692 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:36.259+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:36.259+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:36.259+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:36.398+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:36.398+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:36.398+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:36.398+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:37.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:36.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:36.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:36.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:37.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:37.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:37.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:37.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:37.275+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:37.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:37.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:37.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:37.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:37.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:37.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:37.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:37.443+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:37.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:37.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:37.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:38.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:38.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:38.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:38.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:38.717 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:38.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:38.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:38.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:38.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:38.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:38.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:38.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:38.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:38.481+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:38.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:38.715+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:38.715+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:38.715+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:39.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:39.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:39.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:39.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:39.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:39.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:39.679 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:39.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:39.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:39.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:39.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:39.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:39.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:39.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:39.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:39.677+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:39.677+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:39.677+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:40.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:40.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:40.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:40.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:40.596 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:40.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:40.596 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:40.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:40.597 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:40.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:40.597 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:40.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:40.597 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:40.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:40.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:40.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:40.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:40.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:40.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:40.632+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:41.597 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:41.317+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35764 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:41.338+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:41.338+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:41.338+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:41.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:41.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:41.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:41.598 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:41.508+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:41.595+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:41.595+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:41.595+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:42.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:42.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:42.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:42.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:42.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:42.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:42.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:42.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:42.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:42.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:43.582 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:43.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:43.582 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:43.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:43.582 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:43.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:43.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:43.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:43.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:43.581+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:43.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:43.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:43.603+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:44.575 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:44.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:44.575 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:44.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:44.575 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:44.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:44.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:44.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:44.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:44.574+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:44.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:44.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:44.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:44.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:44.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:44.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:45.610 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:45.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:45.611 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:45.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:45.611 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:45.280+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:45.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:45.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:45.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:45.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:45.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:45.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:45.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:45.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:46.338 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:46.323+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35838 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:46.339 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:46.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:46.339 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:46.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:46.339 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:46.261+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:46.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:46.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:46.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:46.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:46.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:46.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:46.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:47.594 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:47.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:47.594 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:47.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:47.594 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:47.249+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:47.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:47.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:47.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:47.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:47.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:47.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:47.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:47.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:47.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:47.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:48.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:48.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:48.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:48.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:48.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:48.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:48.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:48.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:48.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:48.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:48.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:48.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:49.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:49.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:49.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:49.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:49.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:49.556+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:49.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:49.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:49.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:49.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:50.151+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:50.151+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:50.151+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:50.567+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:50.567+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:50.567+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:50.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:50.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:50.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:50.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:51.452 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:51.328+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35910 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:51.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:51.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:51.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:51.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:51.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:51.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:51.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:51.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:51.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:51.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:51.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:51.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:51.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:51.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:51.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:51.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:51.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:52.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:52.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:52.221+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:52.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:52.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:52.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:52.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:52.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:52.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:52.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:52.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:52.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:52.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:52.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:52.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:52.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:52.738+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:53.558 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:53.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:53.558 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:53.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:53.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:53.256+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:53.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:53.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:53.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:53.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:53.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:53.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:53.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:54.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:54.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:54.581 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:54.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:54.581 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:54.232+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:54.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:54.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:54.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:54.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:54.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:54.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:54.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:55.546 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:55.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:55.547 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:55.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:55.547 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:55.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:55.956 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:55.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:55.956 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:55.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:55.956 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:55.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:55.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:55.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:55.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:55.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:55.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:55.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:55.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:55.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:56.576 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:55:56.334+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 35984 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:55:56.576 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:56.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:56.577 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:56.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:56.577 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:56.242+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:56.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:56.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:56.575+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:56.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:56.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:56.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:56.753+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:57.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:57.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:57.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:57.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:57.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:57.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:57.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:57.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:57.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:58.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:57.802+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:58.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:57.802+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:58.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:57.802+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:58.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:57.802+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:58.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:58.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:58.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:58.776 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:58.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:58.777 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:58.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:58.777 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:58.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:59.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:58.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:59.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:58.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:59.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:58.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:59.140 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:58.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:59.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:59.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:55:59.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:55:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:59.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:59.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:55:59.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:55:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:59.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:59.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:59.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:55:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:55:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:55:59.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:00.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:00.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:00.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:00.555+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:00.555+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:00.555+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:00.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:00.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:00.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:00.755+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:01.461 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:01.339+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36056 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:01.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:01.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:01.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:01.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:01.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:01.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:01.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:01.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:01.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:01.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:02.100 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:01.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:02.100 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:01.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:02.100 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:01.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:02.100 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:01.805+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:02.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:02.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:02.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:02.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:02.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:02.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:02.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:02.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:02.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:03.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:03.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:03.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:02.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:02.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:02.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:02.850+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:03.867 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:03.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:03.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:03.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:03.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:03.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:03.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:03.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:03.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:04.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:03.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:04.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:04.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:04.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:04.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:04.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:04.820 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:04.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:04.820 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:04.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:05.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:04.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:05.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:04.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:05.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:04.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:05.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:04.819+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:05.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:05.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:05.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:05.802 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:05.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:05.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:05.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:05.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:05.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:05.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:05.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:05.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:05.802+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:06.471 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:06.343+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36130 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:06.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:06.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:06.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:06.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:06.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:06.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:06.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:06.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:06.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:07.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:06.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:07.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:06.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:07.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:06.763+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:07.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:06.763+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:07.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:07.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:07.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:07.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:07.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:07.154+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:07.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:07.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:07.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:07.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:07.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:07.715+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:07.715+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:08.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:08.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:08.136+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:08.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:08.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:08.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:08.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:08.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:08.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:08.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:09.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:09.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:09.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:09.520+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:09.520+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:09.520+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:09.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:09.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:09.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:09.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:10.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:10.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:10.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:10.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:10.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:10.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:10.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:10.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:10.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:10.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:11.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:11.348+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36202 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:11.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:11.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:11.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:11.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:11.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:11.123+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:11.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:11.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:11.586+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:12.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:11.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:12.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:11.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:12.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:11.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:12.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:11.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:12.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:12.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:12.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:12.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:12.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:12.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:12.774+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:12.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:12.578+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:12.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:12.578+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:12.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:12.578+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:13.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:13.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:13.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:13.558+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:13.558+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:13.558+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:13.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:13.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:13.759+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:13.759+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:14.388 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:14.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:14.389 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:14.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:14.389 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:14.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:14.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:14.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:14.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:15.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:14.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:15.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:14.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:15.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:14.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:15.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:14.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:15.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:15.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:15.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:15.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:15.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:15.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:15.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:15.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:15.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:16.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:15.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:16.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:15.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:16.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:15.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:16.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:15.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:16.355 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:16.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:16.356 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:16.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:16.356 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:16.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:16.725 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:16.353+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36273 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:16.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:16.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:16.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:16.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:16.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:16.472+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:16.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:16.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:16.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:16.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:16.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:16.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:17.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:17.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:17.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:17.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:17.470 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:17.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:17.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:17.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:17.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:17.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:17.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:17.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:17.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:17.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:18.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:18.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:18.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:18.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:18.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:18.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:18.719+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:18.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:18.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:18.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:19.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:19.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:19.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:19.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:19.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:19.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:19.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:19.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:19.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:19.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:20.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:20.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:20.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:20.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:20.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:20.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:20.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:20.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:20.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:20.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:21.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:21.359+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36339 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:21.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:21.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:21.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:21.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:21.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:21.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:21.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:21.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:21.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:21.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:22.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:22.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:22.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:22.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:22.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:22.522+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:22.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:22.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:22.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:22.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:22.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:22.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:22.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:23.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:23.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:23.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:23.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:23.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:23.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:24.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:23.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:24.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:23.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:24.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:23.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:24.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:23.733+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:24.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:24.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:24.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:24.743 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:24.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:24.743 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:24.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:24.743 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:24.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:25.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:24.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:25.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:24.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:25.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:24.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:25.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:24.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:25.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:25.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:25.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:25.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:25.473 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:25.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:25.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:25.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:25.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:26.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:25.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:26.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:25.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:26.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:25.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:26.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:25.730+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:26.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:26.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:26.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:26.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:26.367 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:26.103+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:26.676 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:26.365+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36406 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:26.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:26.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:26.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:26.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:26.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:26.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:26.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:26.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:26.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:26.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:26.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:27.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:27.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:27.424 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:27.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:27.424 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:27.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:27.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:27.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:27.690 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:27.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:27.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:27.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:27.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:27.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:27.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:27.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:28.447 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:28.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:28.448 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:28.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:28.448 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:28.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:28.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:28.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:28.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:28.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:28.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:28.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:28.658+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:29.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:29.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:29.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:29.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:29.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:29.163+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:29.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:29.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:29.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:29.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:30.400 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:30.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:30.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:30.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:30.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:30.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:30.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:30.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:30.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:31.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:30.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:31.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:30.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:31.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:30.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:31.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:30.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:31.372 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:31.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:31.372 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:31.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:31.372 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:31.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:31.680 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:31.370+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36470 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:31.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:31.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:31.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:31.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:31.681 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:31.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:31.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:31.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:31.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:31.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:32.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:32.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:32.424 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:32.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:32.424 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:32.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:32.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:32.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:32.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:32.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:32.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:32.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:32.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:32.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:32.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:32.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:33.381 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:33.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:33.381 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:33.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:33.381 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:33.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:33.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:33.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:33.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:33.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:33.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:33.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:33.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:33.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:33.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:33.756+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:34.392 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:34.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:34.392 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:34.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:34.393 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:34.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:34.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:34.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:34.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:34.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:34.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:34.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:34.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:34.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:34.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:34.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:35.343 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:35.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:35.343 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:35.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:35.343 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:35.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:35.709 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:35.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:35.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:35.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:35.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:35.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:35.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:35.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:35.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:35.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:36.362 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:36.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:36.362 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:36.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:36.362 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:36.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:36.722 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:36.377+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36534 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:36.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:36.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:36.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:36.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:36.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:36.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:36.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:36.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:36.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:36.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:37.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:37.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:37.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:37.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:37.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:37.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:37.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:37.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:37.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:37.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:37.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:37.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:37.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:37.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:37.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:37.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:37.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:37.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:37.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:37.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:37.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:37.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:37.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:38.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:38.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:38.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:38.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:38.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:38.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:38.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:38.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:38.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:38.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:38.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:38.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:38.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:38.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:39.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:39.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:39.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:39.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:39.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:39.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:39.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:39.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:39.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:39.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:39.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:39.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:39.694+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:40.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:40.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:40.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:40.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:40.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:40.939+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:40.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:40.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:40.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:40.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:41.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:41.383+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36598 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:41.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:41.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:41.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:42.036 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:41.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:42.036 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:41.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:42.036 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:41.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:42.037 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:41.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:42.037 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:41.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:42.037 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:41.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:42.037 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:41.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:42.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:42.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:42.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:42.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:42.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:42.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:42.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:42.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:42.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:42.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:43.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:42.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:43.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:43.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:43.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:43.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:43.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:43.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:43.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:43.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:43.944+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:43.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:43.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:43.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:43.808+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:44.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:44.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:44.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:45.108 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:44.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:45.108 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:44.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:45.108 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:44.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:45.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:44.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:45.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:44.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:45.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:44.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:45.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:44.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:45.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:45.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:45.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:45.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:45.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:45.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:45.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:45.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:45.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:45.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:45.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:45.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:45.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:46.710 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:46.389+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36668 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:46.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:46.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:46.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:46.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:46.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:46.364+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:46.908+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:46.908+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:46.908+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:46.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:46.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:46.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:46.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:47.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:47.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:47.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:47.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:47.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:47.906+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:48.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:47.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:47.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:47.748+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:47.748+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:48.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:48.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:48.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:48.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:48.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:48.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:48.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:48.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:48.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:48.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:49.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:49.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:49.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:49.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:49.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:49.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:50.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:49.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:49.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:49.766+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:49.766+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:50.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:50.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:50.288+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:50.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:50.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:50.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:51.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:50.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:50.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:50.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:50.761+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:51.519 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:51.394+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36738 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:51.519 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:51.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:51.519 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:51.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:51.519 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:51.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:51.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:51.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:51.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:51.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:51.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:51.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:51.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:51.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:51.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:51.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:51.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:52.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:52.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:52.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:53.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:53.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:53.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:53.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:53.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:53.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:53.211 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:52.777+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:53.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:52.777+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:53.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:52.777+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:53.212 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:52.777+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:53.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:53.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:53.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:54.053 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:53.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:54.053 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:53.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:54.053 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:53.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:54.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:53.822+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:54.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:54.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:54.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:54.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:54.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:54.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:54.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:54.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:54.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:54.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:54.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:54.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:55.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:54.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:55.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:54.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:55.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:54.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:55.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:54.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:55.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:55.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:55.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:55.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:55.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:55.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:55.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:55.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:55.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:56.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:55.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:56.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:55.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:56.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:55.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:56.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:55.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:56.400 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:56.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:56.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:56.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:56.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:56.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:56.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:56.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:56.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:56.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:56.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:56.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:56.670 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:56:56.398+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36811 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:56:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:56.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:56.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:56.812+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:56.812+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:57.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:57.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:57.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:57.080+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:57.080+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:57.080+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:57.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:57.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:57.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:57.818+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:58.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:58.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:58.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:58.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:58.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:58.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:58.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:58.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:58.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:59.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:58.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:59.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:58.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:59.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:58.795+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:56:59.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:58.795+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:59.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:59.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:56:59.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:56:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:59.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:59.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:56:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:56:59.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:00.100 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:59.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:00.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:59.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:00.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:59.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:00.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:56:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:56:59.834+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:00.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:00.201+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:00.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:00.201+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:00.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:00.201+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:00.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:00.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:00.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:00.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:00.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:00.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:01.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:00.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:01.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:00.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:01.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:00.831+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:01.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:00.831+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:01.406 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:01.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:01.406 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:01.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:01.406 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:01.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:01.407 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:01.177+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:01.407 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:01.177+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:01.407 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:01.177+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:01.712 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:01.404+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36884 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:01.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:01.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:01.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:01.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:02.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:02.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:02.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:02.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:02.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:02.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:03.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:02.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:03.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:02.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:03.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:02.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:03.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:02.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:03.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:03.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:03.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:03.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:03.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:03.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:04.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:03.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:04.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:03.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:04.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:03.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:04.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:03.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:04.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:04.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:04.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:04.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:04.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:04.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:05.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:04.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:05.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:04.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:05.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:04.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:05.050 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:04.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:05.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:05.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:05.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:05.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:05.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:05.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:06.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:05.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:06.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:05.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:06.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:05.728+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:06.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:05.728+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:06.412 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:06.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:06.412 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:06.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:06.413 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:06.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:06.413 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:06.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:06.413 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:06.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:06.413 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:06.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:06.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:06.410+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 36957 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:07.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:06.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:07.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:06.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:07.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:06.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:07.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:06.729+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:07.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:07.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:07.145+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:07.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:07.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:07.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:07.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:07.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:08.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:07.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:08.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:07.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:08.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:07.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:08.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:07.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:08.110+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:08.110+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:08.110+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:08.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:08.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:08.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:08.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:08.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:08.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:08.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:09.130+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:09.130+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:09.130+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:09.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:09.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:09.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:10.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:09.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:10.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:09.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:10.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:09.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:10.106 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:09.771+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:10.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:10.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:10.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:10.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:10.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:10.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:11.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:10.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:11.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:10.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:11.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:10.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:11.105 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:10.808+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:11.418 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:11.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:11.418 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:11.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:11.418 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:11.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:11.418 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:11.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:11.418 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:11.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:11.418 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:11.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:11.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:11.415+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37030 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:12.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:11.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:12.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:11.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:12.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:11.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:12.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:11.837+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:12.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:12.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:12.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:12.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:12.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:12.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:12.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:12.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:12.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:13.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:12.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:13.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:12.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:13.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:12.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:13.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:12.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:13.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:13.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:13.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:13.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:13.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:13.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:14.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:13.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:14.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:13.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:14.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:13.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:14.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:13.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:14.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:14.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:14.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:14.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:14.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:14.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:14.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:14.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:14.029+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:15.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:14.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:15.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:14.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:15.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:14.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:15.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:14.754+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:15.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:15.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:15.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:15.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:15.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:15.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:15.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:15.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:15.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:15.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:16.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:16.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:16.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:16.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:16.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:16.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:16.423 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:16.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:16.423 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:16.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:16.423 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:16.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:16.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:16.421+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37103 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:17.076 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:16.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:17.077 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:16.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:17.077 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:16.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:17.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:16.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:17.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:16.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:17.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:16.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:17.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:16.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:17.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:17.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:17.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:17.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:17.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:17.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:17.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:17.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:17.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:17.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:17.708+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:17.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:17.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:17.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:18.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:18.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:18.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:18.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:18.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:18.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:18.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:18.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:18.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:18.955 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:18.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:18.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:18.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:18.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:19.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:19.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:19.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:19.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:19.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:19.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:19.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:19.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:19.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:19.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:20.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:20.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:20.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:20.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:20.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:20.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:20.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:20.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:20.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:20.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:20.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:20.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:20.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:20.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:21.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:21.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:21.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:21.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:21.428 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:21.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:21.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:21.426+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37176 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:21.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:21.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:21.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:21.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:21.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:21.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:21.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:22.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:22.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:22.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:22.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:22.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:22.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:22.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:22.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:22.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:22.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:23.052+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:23.052+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:23.052+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:23.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:23.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:23.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:23.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:24.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:24.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:24.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:24.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:24.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:24.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:24.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:24.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:24.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:24.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:24.708+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:24.708+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:25.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:25.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:25.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:25.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:25.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:25.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:26.074 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:25.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:26.075 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:25.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:26.075 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:25.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:26.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:25.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:26.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:25.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:26.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:25.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:26.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:25.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:26.434 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:26.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:26.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:26.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:26.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:26.073+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:26.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:26.433+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37249 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:27.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:26.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:27.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:26.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:27.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:26.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:27.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:26.789+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:27.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:27.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:27.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:27.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:27.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:27.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:27.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:27.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:27.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:28.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:27.780+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:28.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:27.780+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:28.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:28.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:28.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:28.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:28.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:28.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:28.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:28.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:29.026 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:28.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:29.026 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:28.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:29.026 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:28.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:29.026 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:28.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:29.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:29.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:29.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:29.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:29.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:29.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:29.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:29.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:29.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:29.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:29.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:30.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:30.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:30.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:30.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:30.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:30.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:30.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:30.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:30.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:30.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:31.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:31.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:31.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:31.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:31.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:31.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:31.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:31.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:31.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:31.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:31.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:31.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:31.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:31.440+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37322 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:31.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:31.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:31.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:31.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:32.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:32.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:32.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:32.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:32.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:32.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:32.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:32.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:32.674+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:32.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:33.201+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:33.201+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:33.201+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:33.016+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:33.016+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:33.016+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:33.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:33.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:33.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:33.639+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:34.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:34.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:34.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:34.013+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:34.013+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:34.013+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:34.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:34.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:34.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:34.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:34.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:35.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:35.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:35.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:35.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:35.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:35.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:35.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:35.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:35.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:35.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:36.447 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:36.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:36.448 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:36.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:36.448 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:36.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:36.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:36.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:36.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:36.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:36.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:36.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:36.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:36.446+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37395 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:36.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:36.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:36.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:36.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:37.091 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:37.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:37.091 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:37.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:37.091 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:37.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:37.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:37.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:37.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:37.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:37.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:37.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:37.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:37.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:37.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:37.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:38.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:38.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:38.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:38.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:37.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:37.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:37.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:38.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:38.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:38.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:38.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:38.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:38.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:39.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:39.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:39.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:39.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:39.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:38.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:38.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:38.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:39.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:39.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:39.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:39.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:39.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:40.298 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:40.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:40.298 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:40.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:40.298 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:40.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:40.299 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:39.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:40.299 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:39.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:40.299 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:39.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:40.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:40.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:40.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:40.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:41.453 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:40.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:41.454 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:40.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:41.454 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:40.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:41.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:41.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:41.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:41.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:41.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:41.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:41.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:41.452+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37468 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:41.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:41.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:41.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:41.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:42.023+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:42.023+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:42.023+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:42.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:42.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:42.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:42.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:42.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:42.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:42.665+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:43.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:43.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:43.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:43.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:43.085+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:43.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:43.085+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:43.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:43.085+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:43.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:43.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:43.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:43.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:43.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:43.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:43.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:43.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:43.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:43.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:43.972+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:44.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:44.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:44.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:44.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:44.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:44.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:44.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:44.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:44.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:44.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:44.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:45.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:45.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:45.014+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:45.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:45.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:45.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:45.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:45.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:45.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:45.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:45.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:45.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:45.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:45.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:46.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:46.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:46.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:46.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:46.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:46.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:46.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:46.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:46.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:46.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:46.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:46.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:46.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:46.457+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37541 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:46.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:46.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:46.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:46.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:46.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:46.569+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:46.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:46.569+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:47.088 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:47.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:47.088 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:47.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:47.088 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:47.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:47.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:47.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:47.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:47.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:47.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:47.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:47.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:47.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:47.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:47.527+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:48.011+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:48.011+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:48.011+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:48.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:48.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:48.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:48.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:48.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:48.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:48.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:48.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:48.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:48.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:48.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:48.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:48.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:49.139+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:49.139+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:49.139+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:49.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:49.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:49.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:49.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:49.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:49.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:49.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:49.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:49.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:49.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:49.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:50.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:50.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:50.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:50.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:50.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:50.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:50.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:50.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:50.588+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:50.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:50.588+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:50.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:50.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:50.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:51.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:51.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:51.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:51.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:51.463+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37614 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:51.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:51.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:51.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:51.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:51.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:51.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:51.547+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:51.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:51.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:51.954+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:52.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:52.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:52.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:52.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:52.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:52.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:52.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:52.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:52.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:52.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:52.534+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:52.941+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:52.941+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:52.941+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:53.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:53.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:53.150+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:53.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:53.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:53.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:53.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:53.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:53.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:53.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:53.520+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:53.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:53.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:53.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:54.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:54.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:54.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:54.891 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:54.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:54.891 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:54.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:54.891 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:54.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:54.891 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:54.497+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:54.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:54.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:54.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:55.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:55.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:55.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:55.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:55.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:55.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:55.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:55.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:55.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:55.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:55.484+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:56.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:55.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:56.043 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:55.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:56.043 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:55.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:56.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:56.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:56.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:56.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:56.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:56.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:56.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:57:56.468+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37687 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:57:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:56.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:56.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:56.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:56.479+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:57.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:56.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:57.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:56.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:57.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:56.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:57.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:56.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:57.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:56.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:57.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:56.995+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:57.858 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:57.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:57.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:57.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:57.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:57.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:57.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:57.478+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:57.969+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:57.969+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:57.969+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:58.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:57.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:57.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:57.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:58.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:58.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:58.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:58.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:58.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:58.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:58.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:58.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:58.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:58.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:58.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:57:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:58.874+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:58.874+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:58.874+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:57:59.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:59.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:59.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:59.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:59.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:59.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:57:59.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:57:59.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:59.847+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:59.847+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:57:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:57:59.847+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:59.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:59.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:57:59.999+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:00.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:00.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:00.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:00.492+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:01.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:01.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:01.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:00.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:00.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:00.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:01.510 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:01.474+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37760 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:01.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:01.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:01.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:01.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:01.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:01.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:01.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:01.508+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:02.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:01.817+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:02.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:01.817+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:02.214 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:01.817+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:02.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:02.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:02.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:02.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:02.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:02.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:02.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:02.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:02.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:02.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:02.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:02.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:02.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:02.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:03.041 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:02.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:03.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:02.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:03.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:02.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:03.358 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:03.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:03.359 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:03.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:03.359 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:03.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:03.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:03.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:03.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:03.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:03.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:03.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:03.609 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:03.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:03.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:03.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:03.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:04.077+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:04.077+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:04.077+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:04.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:04.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:04.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:04.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:04.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:04.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:04.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:04.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:05.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:04.853+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:05.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:04.853+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:05.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:04.853+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:05.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:05.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:05.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:05.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:05.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:05.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:05.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:05.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:05.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:05.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:05.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:05.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:05.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:06.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:06.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:06.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:06.812 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:06.480+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37833 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:06.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:06.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:06.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:06.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:06.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:06.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:06.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:06.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:07.157 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:06.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:07.157 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:06.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:07.157 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:06.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:07.155+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:07.155+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:07.155+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:07.760 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:07.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:07.760 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:07.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:07.761 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:07.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:07.761 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:07.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:08.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:07.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:08.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:07.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:08.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:07.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:08.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:08.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:08.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:08.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:08.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:08.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:08.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:08.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:08.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:08.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:09.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:09.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:09.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:09.826+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:09.826+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:09.826+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:09.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:09.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:09.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:09.675+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:10.129+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:10.129+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:10.129+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:10.791+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:10.791+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:10.791+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:10.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:10.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:10.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:10.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:11.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:11.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:11.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:11.765 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:11.486+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37906 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:11.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:11.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:11.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:11.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:11.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:11.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:11.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:11.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:12.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:11.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:12.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:11.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:12.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:11.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:12.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:12.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:12.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:12.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:12.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:12.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:12.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:12.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:12.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:12.718+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:13.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:13.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:13.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:13.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:13.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:13.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:13.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:13.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:13.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:13.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:13.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:14.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:14.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:14.096+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:14.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:14.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:14.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:14.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:14.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:14.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:14.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:15.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:15.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:15.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:15.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:15.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:15.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:15.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:15.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:15.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:15.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:16.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:16.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:16.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:16.783 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:16.493+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 37979 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:16.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:16.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:16.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:16.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:16.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:16.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:16.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:16.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:17.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:16.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:17.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:16.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:17.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:16.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:17.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:17.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:17.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:17.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:17.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:17.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:17.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:17.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:17.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:17.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:18.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:18.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:18.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:18.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:18.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:18.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:18.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:18.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:18.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:18.714+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:19.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:19.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:19.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:19.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:19.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:19.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:19.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:19.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:19.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:19.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:19.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:19.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:20.119+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:20.119+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:20.119+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:20.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:20.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:20.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:20.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:20.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:20.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:20.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:21.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:21.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:21.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:21.815 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:21.499+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38052 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:21.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:21.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:21.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:21.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:21.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:21.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:21.816 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:21.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:22.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:21.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:22.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:21.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:22.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:21.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:22.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:22.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:22.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:22.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:22.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:22.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:22.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:22.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:22.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:22.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:23.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:23.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:23.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:23.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:23.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:23.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:23.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:23.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:23.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:23.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:24.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:24.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:24.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:24.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:24.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:24.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:24.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:24.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:24.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:24.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:25.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:25.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:25.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:25.820+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:25.820+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:25.820+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:25.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:25.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:25.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:25.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:26.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:26.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:26.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:26.812 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:26.505+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38125 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:26.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:26.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:26.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:26.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:27.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:26.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:27.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:26.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:27.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:26.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:27.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:27.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:27.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:27.028+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:27.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:27.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:27.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:27.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:27.598+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:28.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:27.831+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:28.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:27.831+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:28.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:27.831+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:28.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:28.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:28.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:28.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:28.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:28.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:28.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:28.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:28.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:28.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:28.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:28.858+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:28.858+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:28.858+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:29.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:29.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:29.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:29.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:29.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:29.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:29.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:29.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:29.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:29.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:29.602+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:29.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:29.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:29.877+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:29.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:29.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:29.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:30.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:30.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:30.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:30.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:30.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:30.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:30.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:30.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:30.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:30.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:30.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:31.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:31.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:31.004+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:31.850 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:31.510+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38198 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:31.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:31.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:31.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:31.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:31.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:31.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:31.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:31.631+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:31.850+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:31.850+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:31.850+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:32.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:32.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:32.048+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:32.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:32.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:32.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:32.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:32.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:32.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:32.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:33.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:33.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:33.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:33.831+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:33.831+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:33.831+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:33.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:33.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:33.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:33.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:33.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:34.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:34.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:34.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:34.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:34.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:34.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:34.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:34.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:34.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:34.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:35.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:35.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:35.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:35.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:35.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:35.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:35.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:35.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:35.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:35.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:36.116+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:36.116+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:36.116+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:36.812 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:36.515+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38271 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:36.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:36.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:36.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:36.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:36.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:37.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:36.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:37.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:36.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:37.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:36.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:37.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:37.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:37.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:37.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:37.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:37.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:37.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:37.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:37.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:37.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:37.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:38.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:37.841+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:38.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:37.841+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:38.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:37.841+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:38.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:38.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:38.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:38.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:38.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:38.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:38.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:38.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:38.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:38.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:38.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:39.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:38.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:39.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:38.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:39.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:38.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:39.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:39.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:39.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:39.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:39.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:39.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:39.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:39.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:39.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:39.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:39.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:40.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:40.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:40.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:39.900+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:39.900+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:39.900+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:40.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:40.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:40.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:40.482+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:40.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:40.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:40.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:41.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:41.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:41.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:41.844 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:41.520+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38344 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:41.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:41.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:41.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:41.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:41.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:41.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:41.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:41.516+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:41.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:41.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:41.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:42.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:42.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:42.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:42.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:42.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:42.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:42.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:42.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:42.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:42.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:42.497+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:42.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:42.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:42.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:43.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:43.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:43.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:43.900 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:43.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:43.900 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:43.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:43.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:43.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:43.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:43.535+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:43.968+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:43.968+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:43.968+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:44.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:43.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:44.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:43.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:44.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:43.898+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:44.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:44.555+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:44.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:44.555+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:44.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:44.555+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:44.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:44.555+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:44.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:44.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:44.993+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:44.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:44.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:44.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:45.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:45.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:45.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:45.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:45.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:45.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:45.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:45.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:46.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:46.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:46.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:45.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:45.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:45.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:46.814 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:46.527+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38417 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:46.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:46.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:46.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:46.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:47.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:47.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:47.038+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:46.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:46.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:46.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:47.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:47.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:47.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:47.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:48.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:48.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:48.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:48.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:48.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:48.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:48.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:48.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:48.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:48.585+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:49.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:49.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:49.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:49.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:49.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:49.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:49.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:49.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:49.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:49.630+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:50.304 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:49.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:50.304 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:49.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:50.305 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:49.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:50.305 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:50.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:50.305 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:50.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:50.305 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:50.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:50.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:50.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:50.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:50.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:50.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:50.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:50.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:50.639+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:51.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:51.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:51.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:50.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:50.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:50.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:51.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:51.533+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38490 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:51.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:51.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:51.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:51.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:52.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:52.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:52.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:52.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:52.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:52.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:52.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:52.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:52.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:52.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:52.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:52.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:52.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:52.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:52.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:52.966 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:52.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:53.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:53.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:53.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:52.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:52.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:52.964+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:53.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:53.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:53.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:53.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:53.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:53.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:53.572+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:54.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:54.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:54.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:53.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:53.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:53.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:54.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:54.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:54.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:54.581+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:55.080+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:55.080+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:55.080+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:55.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:55.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:55.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:55.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:55.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:55.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:55.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:55.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:55.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:55.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:55.622+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:56.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:56.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:56.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:55.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:56.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:55.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:56.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:55.962+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:56.812 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:58:56.539+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38563 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:58:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:56.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:56.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:56.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:56.658+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:57.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:56.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:57.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:56.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:57.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:56.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:57.106+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:57.106+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:57.106+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:57.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:57.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:57.636+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:57.636+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:58.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:58.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:58.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:57.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:57.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:57.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:58.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:58.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:58.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:58.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:58.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:58.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:58.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:58.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:59.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:59.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:59.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:58:59.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:58:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:58.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:58.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:58.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:58:59.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:59.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:59.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:59.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:59.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:59.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:58:59.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:58:59.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:00.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:00.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:00.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:59.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:59.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:58:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:58:59.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:00.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:00.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:00.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:00.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:01.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:01.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:01.067+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:00.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:00.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:00.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:01.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:01.546+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38636 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:01.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:01.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:01.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:01.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:02.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:02.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:02.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:02.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:02.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:02.001+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:02.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:02.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:02.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:02.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:03.348 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:03.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:03.348 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:03.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:03.348 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:03.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:03.349 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:03.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:03.349 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:03.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:03.349 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:03.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:03.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:03.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:03.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:03.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:03.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:03.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:03.967 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:03.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:03.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:03.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:03.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:04.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:04.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:04.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:04.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:04.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:04.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:04.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:04.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:04.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:04.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:04.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:04.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:04.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:04.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:04.954+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:05.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:05.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:05.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:05.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:05.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:05.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:05.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:05.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:05.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:05.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:06.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:06.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:06.420 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:06.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:06.421 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:06.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:06.677 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:06.552+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38709 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:06.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:06.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:06.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:06.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:06.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:06.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:06.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:06.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:06.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:06.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:07.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:07.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:07.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:07.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:07.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:07.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:07.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:07.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:07.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:07.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:07.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:08.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:08.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:08.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:07.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:07.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:07.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:08.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:08.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:08.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:08.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:08.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:08.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:08.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:08.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:09.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:09.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:09.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:09.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:09.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:09.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:08.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:08.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:08.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:09.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:09.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:09.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:09.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:09.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:09.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:09.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:09.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:10.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:10.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:10.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:10.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:10.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:09.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:10.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:09.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:10.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:09.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:10.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:10.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:10.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:10.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:10.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:10.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:10.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:11.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:11.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:11.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:11.077+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:11.900 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:11.557+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38782 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:11.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:11.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:11.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:11.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:11.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:11.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:11.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:11.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:12.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:12.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:12.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:11.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:11.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:11.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:12.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:12.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:12.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:12.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:12.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:12.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:12.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:12.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:13.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:12.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:13.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:12.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:13.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:12.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:13.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:13.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:13.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:13.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:13.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:13.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:13.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:13.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:13.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:13.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:13.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:14.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:13.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:14.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:13.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:14.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:13.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:14.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:14.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:14.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:14.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:14.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:14.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:14.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:14.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:14.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:14.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:14.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:15.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:15.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:15.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:15.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:15.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:15.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:15.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:15.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:15.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:15.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:16.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:16.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:16.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:16.921 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:16.564+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38855 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:16.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:16.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:16.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:16.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:16.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:16.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:16.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:16.661+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:16.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:16.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:16.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:17.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:17.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:17.645 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:17.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:17.645 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:17.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:17.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:17.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:17.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:17.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:17.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:17.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:17.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:18.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:18.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:18.687 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:18.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:18.687 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:18.245+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:18.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:18.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:18.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:18.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:18.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:18.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:18.908+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:19.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:19.268+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:19.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:19.268+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:19.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:19.268+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:19.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:19.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:19.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:19.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:19.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:19.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:19.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:19.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:19.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:19.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:19.943+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:20.651 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:20.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:20.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:20.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:20.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:20.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:20.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:20.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:20.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:20.650+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:21.267 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:20.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:21.267 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:20.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:21.267 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:20.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:21.571 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:21.266+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:21.572 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:21.266+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:21.572 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:21.266+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:21.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:21.570+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 38928 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:21.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:21.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:21.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:21.666+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:22.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:22.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:22.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:22.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:22.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:22.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:22.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:22.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:22.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:22.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:22.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:22.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:22.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:23.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:23.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:23.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:23.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:23.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:23.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:23.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:23.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:23.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:23.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:24.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:24.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:24.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:24.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:24.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:24.204+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:24.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:23.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:24.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:23.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:24.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:23.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:24.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:24.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:24.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:24.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:24.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:25.203+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:25.203+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:25.203+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:24.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:24.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:24.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:25.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:25.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:25.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:25.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:26.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:26.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:26.186+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:26.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:26.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:26.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:26.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:26.576+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39001 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:26.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:26.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:26.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:26.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:26.987+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:26.987+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:26.987+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:27.211+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:27.211+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:27.211+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:27.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:27.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:27.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:27.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:27.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:27.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:27.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:27.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:27.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:27.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:27.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:28.578 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:28.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:28.579 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:28.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:28.579 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:28.235+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:28.577+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:28.577+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:28.577+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:28.577+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:29.276 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:28.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:29.276 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:28.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:29.276 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:28.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:29.609 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:29.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:29.609 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:29.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:29.609 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:29.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:29.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:29.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:29.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:29.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:29.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:29.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:29.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:29.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:29.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:29.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:29.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:30.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:30.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:30.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:30.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:30.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:30.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:30.956 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:30.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:30.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:30.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:30.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:30.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:30.957 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:30.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:30.955+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:30.955+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:30.955+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:31.585 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:31.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:31.586 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:31.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:31.586 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:31.241+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:31.950 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:31.583+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39074 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:31.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:31.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:31.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:31.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:31.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:31.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:31.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:31.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:31.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:31.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:31.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:32.659 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:32.250+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:32.659 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:32.250+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:32.659 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:32.250+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:32.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:32.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:32.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:32.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:32.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:32.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:32.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:32.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:32.914+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:32.914+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:32.914+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:33.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:33.267+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:33.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:33.267+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:33.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:33.267+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:33.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:33.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:33.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:33.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:33.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:33.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:33.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:33.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:34.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:34.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:34.271+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:34.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:34.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:34.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:34.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:34.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:34.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:34.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:34.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:35.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:35.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:35.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:35.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:35.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:35.255+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:35.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:35.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:35.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:35.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:35.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:35.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:35.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:35.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:36.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:36.303+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:36.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:36.303+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:36.592 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:36.303+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:36.863 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:36.590+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39147 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:36.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:36.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:36.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:36.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:36.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:36.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:36.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:36.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:36.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:36.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:36.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:37.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:37.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:37.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:37.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:37.674 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:37.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:37.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:37.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:37.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:37.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:37.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:37.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:37.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:38.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:38.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:38.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:38.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:38.642 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:38.291+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:38.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:38.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:38.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:38.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:38.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:38.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:38.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:39.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:39.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:39.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:39.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:39.607 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:39.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:39.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:39.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:39.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:39.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:39.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:39.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:39.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:40.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:40.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:40.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:40.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:40.597 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:40.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:40.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:40.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:40.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:40.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:40.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:40.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:40.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:40.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:41.724 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:41.596+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39220 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:41.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:41.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:41.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:41.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:41.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:41.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:41.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:41.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:41.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:41.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:41.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:41.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:41.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:41.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:41.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:41.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:41.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:41.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:42.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:42.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:42.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:42.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:42.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:42.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:42.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:43.052 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:42.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:43.052 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:42.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:43.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:42.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:43.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:43.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:43.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:43.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:43.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:43.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:43.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:43.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:43.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:43.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:44.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:44.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:44.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:44.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:44.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:44.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:44.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:44.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:44.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:44.794 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:44.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:44.794 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:44.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:44.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:44.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:44.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:45.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:45.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:45.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:45.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:45.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:45.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:45.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:45.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:45.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:45.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:45.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:45.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:45.760+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:46.614 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:46.602+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39293 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:46.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:46.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:46.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:46.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:46.615 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:46.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:46.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:46.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:46.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:46.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:46.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:46.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:46.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:46.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:47.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:47.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:47.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:47.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:47.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:47.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:47.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:47.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:47.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:47.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:48.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:48.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:48.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:48.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:48.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:48.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:48.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:48.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:48.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:48.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:49.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:49.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:49.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:49.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:49.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:49.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:49.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:49.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:49.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:49.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:49.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:49.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:49.744+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:50.617 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:50.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:50.618 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:50.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:50.618 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:50.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:50.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:50.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:50.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:50.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:50.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:50.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:50.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:51.777 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:51.610+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39365 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:51.777 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:51.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:51.778 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:51.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:51.778 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:51.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:51.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:51.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:51.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:51.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:51.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:51.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:51.778 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:51.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:51.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:51.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:51.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:52.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:52.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:52.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:52.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:52.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:52.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:52.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:53.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:52.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:53.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:52.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:53.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:52.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:53.858 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:53.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:53.858 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:53.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:53.858 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:53.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:53.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:53.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:53.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:53.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:53.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:53.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:53.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:53.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:53.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:53.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:53.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:54.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:54.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:54.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:54.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:54.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:54.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:54.852 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:54.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:54.852 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:54.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:54.852 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:54.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:54.852 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:54.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:54.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:54.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:54.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:55.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:55.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:55.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:55.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:55.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:55.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:55.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:55.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:55.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:55.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:55.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:55.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:55.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:55.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:56.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T19:59:56.616+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39439 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T19:59:56.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:56.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:56.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:56.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:56.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:56.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:56.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:56.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:56.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:56.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:56.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:56.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:56.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:56.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:56.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:56.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:56.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:56.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:56.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:56.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:57.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:57.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:57.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:57.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:57.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:57.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:57.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:57.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:57.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:57.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:58.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:58.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:58.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:58.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:58.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:58.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:58.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:58.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:58.641+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:58.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:58.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:58.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:58.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:58.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:58.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:59.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:59.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:59.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:59.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:59.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T19:59:59.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T19:59:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:59.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:59.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T19:59:59.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T19:59:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:59.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:59.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:59.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T19:59:59.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 19:59:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T19:59:59.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:00.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:00.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:00.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:00.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:00.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:00.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:00.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:00.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:00.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:00.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:00.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:00.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:00.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:00.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:00.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:01.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:01.622+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39511 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:01.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:01.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:01.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:01.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:01.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:01.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:01.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:01.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:01.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:01.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:01.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:01.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:01.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:01.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:01.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:01.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:02.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:02.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:02.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:02.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:02.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:02.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:02.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:02.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:02.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:02.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:03.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:03.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:03.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:03.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:03.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:03.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:03.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:03.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:03.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:03.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:03.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:03.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:03.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:04.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:04.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:04.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:04.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:04.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:04.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:04.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:04.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:04.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:04.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:04.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:04.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:04.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:04.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:05.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:05.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:05.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:05.637+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:05.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:05.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:05.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:05.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:05.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:05.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:05.718+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:06.880 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:06.628+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39585 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:06.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:06.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:06.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:06.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:06.881 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:06.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:06.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:06.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:06.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:06.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:06.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:06.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:06.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:06.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:06.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:06.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:06.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:06.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:06.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:06.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:07.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:07.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:07.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:07.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:07.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:07.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:07.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:07.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:07.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:07.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:07.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:07.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:08.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:08.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:08.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:08.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:08.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:08.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:08.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:08.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:08.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:08.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:08.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:08.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:08.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:08.658+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:08.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:08.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:08.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:09.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:09.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:09.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:09.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:09.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:09.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:09.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:09.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:09.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:09.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:10.859 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:10.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:10.859 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:10.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:10.859 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:10.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:10.859 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:10.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:10.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:10.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:10.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:10.687+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:10.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:10.687+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:10.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:10.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:11.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:10.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:11.867 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:11.635+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39657 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:11.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:11.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:11.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:11.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:11.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:11.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:11.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:11.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:11.868 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:11.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:11.868 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:11.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:11.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:11.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:12.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:11.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:12.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:11.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:12.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:11.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:12.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:12.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:12.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:12.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:12.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:12.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:12.884 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:12.679+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:12.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:12.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:12.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:12.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:12.885 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:12.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:12.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:12.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:13.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:12.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:13.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:13.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:13.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:13.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:13.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:13.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:13.865 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:13.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:13.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:13.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:13.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:13.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:13.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:13.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:13.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:13.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:14.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:13.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:14.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:14.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:14.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:14.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:14.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:14.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:14.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:14.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:14.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:14.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:14.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:14.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:14.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:14.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:14.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:14.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:14.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:15.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:15.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:15.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:15.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:15.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:15.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:15.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:15.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:15.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:15.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:15.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:15.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:15.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:15.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:15.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:15.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:15.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:16.812 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:16.639+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39731 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:16.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:16.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:16.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:16.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:16.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:16.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:16.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:16.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:16.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:16.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:17.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:16.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:17.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:17.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:17.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:17.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:17.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:17.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:17.907 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:17.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:17.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:17.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:17.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:17.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:17.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:17.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:17.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:17.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:17.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:18.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:18.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:18.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:18.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:18.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:18.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:18.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:18.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:18.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:18.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:19.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:19.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:19.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:19.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:19.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:19.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:19.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:19.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:19.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:19.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:19.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:19.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:19.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:19.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:20.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:20.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:20.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:20.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:20.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:20.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:20.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:20.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:20.976 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:20.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:20.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:20.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:20.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:20.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:20.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:20.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:20.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:21.951 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:21.645+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39803 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:21.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:21.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:21.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:21.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:21.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:21.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:21.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:21.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:21.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:21.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:21.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:21.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:21.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:21.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:21.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:21.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:21.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:22.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:22.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:22.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:22.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:22.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:22.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:22.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:22.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:22.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:22.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:22.945 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:22.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:22.945 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:22.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:23.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:22.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:23.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:22.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:23.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:22.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:23.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:23.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:23.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:23.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:23.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:23.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:23.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:23.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:23.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:23.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:24.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:24.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:24.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:24.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:24.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:24.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:24.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:24.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:24.770+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:24.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:24.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:24.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:25.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:25.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:25.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:25.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:25.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:25.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:25.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:26.292 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:25.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:26.292 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:25.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:26.293 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:25.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:26.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:26.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:26.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:26.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:26.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:26.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:26.968 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:26.650+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39877 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:26.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:26.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:26.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:26.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:26.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:26.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:26.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:26.719+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:26.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:26.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:26.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:27.748 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:27.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:27.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:27.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:27.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:27.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:27.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:27.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:27.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:27.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:28.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:27.747+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:28.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:28.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:28.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:28.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:28.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:28.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:28.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:28.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:28.717+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:28.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:28.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:28.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:28.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:28.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:29.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:29.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:29.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:29.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:29.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:29.694+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:29.694+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:30.309 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:29.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:30.309 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:29.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:30.309 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:29.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:30.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:30.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:30.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:30.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:30.559 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:30.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:30.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:30.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:30.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:30.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:30.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:31.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:31.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:31.037+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:31.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:31.656+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 39949 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:31.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:31.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:31.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:31.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:31.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:31.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:31.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:31.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:31.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:31.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:32.436 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:32.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:32.436 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:32.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:32.436 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:32.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:32.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:32.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:32.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:32.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:33.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:32.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:33.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:32.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:33.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:32.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:33.078 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:32.736+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:33.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:33.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:33.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:33.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:33.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:33.076+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:33.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:33.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:33.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:34.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:33.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:34.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:33.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:34.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:33.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:34.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:33.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:34.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:34.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:34.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:34.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:34.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:34.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:34.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:34.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:34.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:34.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:34.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:34.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:35.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:35.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:35.020+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:35.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:35.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:35.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:35.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:35.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:35.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:35.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:35.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:36.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:36.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:36.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:36.785 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:36.662+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40023 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:36.785 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:36.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:36.785 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:36.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:36.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:36.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:36.786 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:36.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:36.786 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:36.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:36.786 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:36.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:36.786 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:36.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:37.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:37.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:37.011+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:37.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:37.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:37.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:37.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:37.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:37.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:37.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:37.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:37.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:38.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:38.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:38.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:38.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:38.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:38.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:38.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:38.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:38.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:38.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:39.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:39.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:39.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:39.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:39.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:39.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:39.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:39.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:39.718+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:39.718+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:40.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:40.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:40.113+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:40.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:40.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:40.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:40.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:40.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:40.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:40.701+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:41.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:41.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:41.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:41.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:41.669+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40095 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:41.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:41.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:41.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:41.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:41.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:41.688+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:41.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:42.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:42.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:42.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:42.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:42.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:42.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:42.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:42.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:42.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:42.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:43.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:43.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:43.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:43.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:43.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:43.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:43.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:43.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:43.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:43.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:43.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:44.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:44.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:44.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:44.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:44.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:44.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:44.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:44.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:44.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:44.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:45.187+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:45.187+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:45.187+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:45.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:45.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:45.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:45.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:45.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:45.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:45.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:46.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:46.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:46.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:46.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:46.675+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40168 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:46.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:46.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:46.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:46.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:46.741+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:46.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:46.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:46.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:46.706+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:47.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:47.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:47.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:47.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:47.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:47.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:47.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:47.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:47.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:47.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:47.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:48.600 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:48.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:48.600 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:48.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:48.600 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:48.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:48.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:48.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:48.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:48.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:48.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:48.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:48.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:49.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:49.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:49.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:49.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:49.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:49.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:49.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:49.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:49.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:49.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:50.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:50.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:50.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:50.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:50.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:50.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:50.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:50.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:50.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:50.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:50.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:50.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:50.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:50.755+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:51.682 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:51.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:51.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:51.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:51.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:51.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:51.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:51.681+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40240 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:51.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:51.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:51.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:51.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:51.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:51.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:51.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:51.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:51.733+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:52.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:52.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:52.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:52.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:52.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:52.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:52.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:52.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:52.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:52.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:52.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:52.768+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:52.768+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:53.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:53.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:53.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:53.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:53.642 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:53.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:53.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:53.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:53.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:53.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:53.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:53.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:53.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:53.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:53.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:53.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:53.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:53.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:53.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:53.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:54.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:54.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:54.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:54.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:54.674 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:54.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:54.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:54.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:54.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:54.758+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:54.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:54.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:54.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:54.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:54.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:54.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:55.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:55.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:55.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:55.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:55.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:55.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:55.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:55.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:55.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:55.714+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:55.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:55.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:55.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:56.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:56.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:56.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:56.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:56.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:56.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:56.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:00:56.687+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40314 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:56.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:56.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:56.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:56.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:56.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:56.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:56.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:56.654+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:57.651 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:57.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:57.651 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:57.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:57.651 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:57.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:57.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:57.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:57.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:57.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:57.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:57.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:57.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:57.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:57.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:57.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:58.608 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:58.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:58.608 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:58.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:58.609 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:58.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:58.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:58.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:58.792+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:58.792+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:58.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:58.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:58.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:58.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:58.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:58.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:59.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:59.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:59.634 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:59.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:59.634 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:00:59.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:00:59.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:59.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:59.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:59.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:59.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:00:59.632+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:00:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:59.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:59.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:59.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:00:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:00:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:00:59.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:00.595 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:00.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:00.596 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:00.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:00.596 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:00.311+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:00.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:00.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:00.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:00.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:00.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:00.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:00.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:01.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:01.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:01.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:01.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:01.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:01.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:01.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:01.693+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40386 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:01.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:01.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:01.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:01.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:01.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:01.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:01.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:01.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:01.819+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:01.819+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:02.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:02.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:02.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:02.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:02.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:02.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:02.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:03.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:02.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:03.156 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:02.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:03.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:02.818+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:03.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:02.818+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:03.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:03.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:03.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:03.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:03.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:03.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:03.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:03.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:03.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:03.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:03.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:03.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:03.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:04.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:04.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:04.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:04.327+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:04.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:04.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:04.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:04.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:04.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:04.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:04.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:04.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:05.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:05.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:05.359+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:05.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:05.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:05.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:05.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:05.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:06.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:05.789+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:05.789+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:06.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:06.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:06.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:06.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:06.702 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:06.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:06.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:06.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:06.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:06.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:06.703 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:06.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:06.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:06.700+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40460 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:06.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:06.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:06.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:06.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:06.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:06.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:07.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:07.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:07.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:07.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:07.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:07.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:08.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:07.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:07.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:07.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:07.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:08.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:08.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:08.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:08.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:08.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:08.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:08.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:08.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:08.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:08.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:09.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:09.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:09.421+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:09.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:09.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:09.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:09.294+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:09.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:09.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:09.850+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:09.850+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:10.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:10.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:10.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:10.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:10.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:10.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:10.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:10.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:11.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:10.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:10.880+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:11.709 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:11.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:11.709 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:11.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:11.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:11.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:11.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:11.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:11.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:11.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:11.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:11.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:11.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:11.707+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40532 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:11.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:11.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:11.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:11.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:12.253+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:12.253+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:12.253+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:12.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:12.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:12.485+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:12.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:12.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:12.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:12.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:13.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:13.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:13.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:13.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:13.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:13.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:13.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:14.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:13.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:13.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:13.923+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:14.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:14.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:14.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:14.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:14.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:14.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:14.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:14.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:14.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:15.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:14.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:14.936+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:15.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:15.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:15.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:15.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:15.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:15.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:16.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:15.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:15.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:15.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:15.972+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:16.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:16.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:16.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:16.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:16.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:16.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:16.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:16.716 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:16.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:16.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:16.713+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40606 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:17.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:16.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:17.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:16.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:17.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:16.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:17.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:16.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:17.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:17.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:17.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:17.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:17.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:17.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:17.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:18.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:17.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:18.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:17.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:18.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:17.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:18.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:17.982+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:18.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:18.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:18.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:18.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:18.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:18.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:18.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:18.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:19.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:18.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:18.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:19.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:19.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:19.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:19.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:19.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:19.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:19.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:19.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:19.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:20.313 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:20.313 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:20.313 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.001+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:20.313 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.001+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:20.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:20.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:20.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:20.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:20.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:20.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:20.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:21.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:21.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:21.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:21.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:20.997+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:21.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:21.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:21.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:21.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:21.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:21.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:21.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:21.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:21.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:21.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:21.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:21.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:21.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:21.719+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40678 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:22.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:22.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:22.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:22.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:22.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:22.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:22.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:22.040+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:22.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:22.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:22.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:22.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:22.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:22.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:23.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:23.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:23.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:23.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:23.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:23.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:23.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:23.048+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:23.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:23.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:23.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:23.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:23.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:23.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:24.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:24.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:24.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:24.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.041+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:24.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:24.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:24.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:24.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:24.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:24.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:24.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:24.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:24.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:25.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:25.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:25.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:25.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:24.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:25.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:25.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:25.520+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:25.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:25.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:25.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:25.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:25.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:25.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:26.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:25.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:26.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:25.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:26.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:25.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:26.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:25.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:26.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:26.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:26.445+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:26.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:26.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:26.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:26.986 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:26.725+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40752 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:27.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:26.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:27.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:26.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:27.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:26.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:27.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:26.984+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:27.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:27.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:27.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:27.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:27.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:27.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:28.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:28.935 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:28.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:28.935 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:28.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:28.936 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:28.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:28.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:28.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:28.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:28.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:28.936 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:28.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:28.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:29.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:29.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:29.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:29.511+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:29.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:29.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:29.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:29.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:29.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:29.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:29.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:29.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:30.313 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:30.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:30.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:30.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:30.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:30.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:30.314 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:30.010+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:30.586 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:30.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:30.586 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:30.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:30.586 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:30.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:30.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:30.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:30.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:31.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:31.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:31.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:31.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:31.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:31.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:31.025+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:31.745 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:31.733+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40823 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:31.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:31.564+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:31.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:31.564+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:31.746 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:31.564+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:31.747 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:31.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:31.747 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:31.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:31.747 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:31.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:32.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:32.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:32.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:32.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:32.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:32.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:32.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:32.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:33.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:32.998+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:33.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:33.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:33.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:33.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:33.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:33.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:33.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:33.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:33.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:33.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:33.877 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:33.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:33.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:34.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:33.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:33.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:34.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:33.952+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:34.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:34.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:34.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:34.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:34.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:34.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:34.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:34.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:34.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:34.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:34.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:34.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:34.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:34.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:35.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:34.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:35.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:34.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:35.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:35.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:35.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:35.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:35.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:35.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:35.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:35.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:35.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:35.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:35.882 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:35.491+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:35.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:35.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:35.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:36.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:35.880+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:36.752 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:36.740+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40897 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:36.752 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:36.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:36.752 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:36.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:36.752 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:36.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:36.753 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:36.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:36.753 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:36.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:36.753 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:36.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:37.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:36.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:36.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:36.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:37.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:36.890+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:37.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:37.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:37.885 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:37.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:37.885 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:37.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:37.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:37.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:37.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:37.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:37.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:37.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:37.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:38.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:37.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:37.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:37.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:38.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:38.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:38.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:38.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:38.932 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:38.529+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:38.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:38.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:38.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:38.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:38.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:38.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:38.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:38.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:39.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:38.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:39.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:38.931+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:39.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:39.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:39.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:39.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:39.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:39.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:39.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:39.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:39.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:39.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:39.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:39.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:40.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:39.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:40.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:40.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:40.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:40.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:40.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:40.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:40.526+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:41.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:40.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:40.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:40.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:40.917+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:41.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:41.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:41.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:41.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:41.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:41.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:41.562+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:42.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:41.746+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 40969 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:42.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:41.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:41.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:41.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:41.952+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:42.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:42.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:42.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:42.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:42.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:42.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:42.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:42.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:42.546+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:43.433 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:42.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:43.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:42.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:43.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:42.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:43.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:42.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:43.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:43.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:43.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:43.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:43.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:43.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:43.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:43.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:43.593+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:44.463 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:44.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:44.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:44.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:44.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:44.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:44.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:44.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:44.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:44.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:44.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:44.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:44.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:44.631+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:45.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:45.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:45.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:45.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:45.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:45.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:45.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:45.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:45.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:45.623+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:46.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:46.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:46.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:46.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:46.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:46.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:46.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:46.040+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:46.754 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:46.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:46.754 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:46.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:46.755 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:46.643+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:46.755 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:46.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:46.755 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:46.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:46.755 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:46.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:47.059 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:46.753+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41043 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:47.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:47.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:47.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:47.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:47.452 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:47.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:47.452 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:47.058+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:47.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:47.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:47.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:47.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:47.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:47.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:47.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:48.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:48.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:48.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:48.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:48.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:48.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:48.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:48.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:48.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:48.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:48.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:48.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:48.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:48.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:48.614+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:49.452 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:49.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:49.452 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:49.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:49.452 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:49.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:49.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:49.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:49.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:49.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:49.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:49.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:49.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:49.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:50.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:50.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:50.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:50.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:50.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:50.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:50.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:50.073+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:50.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:50.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:50.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:50.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:50.648+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:50.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:50.648+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:50.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:50.648+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:51.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:51.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:51.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:51.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:51.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:51.073+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:51.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:51.073+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:51.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:51.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:51.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:51.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:51.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:51.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:51.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:51.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:51.613+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:52.120 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:51.758+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41115 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:52.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:52.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:52.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:52.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:52.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:52.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:52.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:52.118+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:52.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:52.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:52.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:52.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:52.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:52.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:52.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:52.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:53.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:53.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:53.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:53.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:53.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:53.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:53.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:53.095+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:53.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:53.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:53.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:53.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:53.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:53.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:53.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:54.389 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:54.063+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:54.389 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:54.063+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:54.389 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:54.063+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:54.389 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:54.063+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:54.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:54.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:54.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:54.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:54.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:54.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:54.711+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:54.711+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:54.711+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:55.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:55.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:55.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:55.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:55.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:55.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:55.359 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:55.086+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:55.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:55.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:55.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:55.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:55.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:55.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:56.107 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:55.739+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:56.107 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:55.739+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:56.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:55.739+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:56.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:56.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:56.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:56.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:56.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:56.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:56.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:56.105+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:56.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:56.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:56.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:57.147 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:01:56.764+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41188 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:01:57.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:56.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:57.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:56.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:57.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:56.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:57.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:57.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:57.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:57.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:57.145+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:57.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:57.145+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:57.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:57.145+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:57.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:57.145+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:58.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:57.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:58.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:57.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:58.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:57.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:58.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:58.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:58.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:58.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:58.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:58.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:58.191+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:59.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:58.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:59.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:58.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:59.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:58.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:01:59.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:59.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:59.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:59.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:59.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:59.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:01:59.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:01:59.147+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:59.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:59.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:01:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:01:59.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:00.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:59.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:00.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:59.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:00.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:01:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:01:59.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:00.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:00.121+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:00.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:00.121+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:00.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:00.121+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:00.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:00.121+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:00.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:00.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:00.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:00.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:00.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:00.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:01.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:00.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:01.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:00.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:01.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:00.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:01.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:01.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:01.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:01.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:01.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:01.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:01.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:01.782 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:01.771+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41260 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:02.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:01.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:02.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:01.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:02.103 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:01.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:02.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:02.100+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:02.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:02.100+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:02.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:02.100+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:02.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:02.100+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:02.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:02.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:02.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:02.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:02.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:02.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:03.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:03.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:03.079+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:03.079+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:03.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:03.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:03.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:04.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:03.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:04.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:03.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:04.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:03.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:04.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:04.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:04.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:04.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:04.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:04.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:04.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:04.058+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:04.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:04.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:04.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:05.079 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:04.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:05.079 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:04.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:05.079 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:04.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:05.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:05.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:05.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:05.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:05.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:05.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:05.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:05.077+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:05.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:05.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:05.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:05.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:06.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:05.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:06.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:05.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:06.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.042+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:06.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.042+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:06.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:06.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:06.429+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:07.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:06.778+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41335 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:06.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:06.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:06.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:06.996+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:07.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:07.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:07.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:07.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:08.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:07.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:07.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:07.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:07.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:07.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:07.987+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:08.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:08.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:08.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:08.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:08.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:09.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:08.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:08.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:08.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:08.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:08.998+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:09.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:09.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:09.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:09.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:10.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:09.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:10.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:09.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:10.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:09.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:09.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:09.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:09.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:09.956+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:10.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:10.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:10.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:11.001 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:10.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:11.001 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:10.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:11.001 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:10.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:10.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:10.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:10.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:10.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:11.787 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:11.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:11.787 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:11.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:11.787 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:11.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:12.039 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:11.784+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41407 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:12.039 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:11.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:12.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:11.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:12.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:11.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:12.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:12.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:12.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:12.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:12.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:12.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:12.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:12.037+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:12.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:12.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:12.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:13.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:13.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:13.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:13.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:13.040+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:12.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:12.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:12.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:13.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:13.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:13.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:13.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:13.749 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:13.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:14.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:13.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:14.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:13.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:14.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:13.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:14.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:14.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:14.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:14.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:14.321 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:14.019+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:14.321 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:14.019+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:14.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:14.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:14.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:14.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:14.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:14.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:14.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:15.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:15.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:15.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:15.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:15.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:15.055+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:15.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:15.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:15.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:16.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:15.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:16.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:15.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:16.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:15.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:16.100+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:16.100+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:16.100+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:16.100+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:16.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:16.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:16.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:16.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:16.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:16.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:17.097 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:16.791+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41481 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:17.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:16.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:17.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:16.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:17.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:16.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:17.096+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:17.096+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:17.096+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:17.096+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:17.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:17.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:17.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:17.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:17.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:17.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:17.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:17.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:17.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:18.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:18.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:18.094+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:18.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:18.094+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:18.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:18.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:18.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:18.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:18.817 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:18.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:18.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:18.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:18.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:19.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:19.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:19.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:19.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:19.056+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:19.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:19.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:19.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:20.100 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:19.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:20.100 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:19.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:20.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:19.773+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:20.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:20.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:20.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:20.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:20.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:20.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:20.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:20.099+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:20.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:20.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:20.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:21.081 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:20.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:21.081 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:20.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:21.081 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:20.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:21.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:21.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:21.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:21.080+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:21.800 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:21.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:21.800 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:21.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:21.800 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:21.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:21.800 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:21.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:21.801 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:21.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:21.801 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:21.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:22.073 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:21.798+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41553 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:22.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:22.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:22.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:22.072+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:22.758 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:22.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:22.758 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:22.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:22.758 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:22.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:23.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:22.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:23.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:22.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:23.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:22.757+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:23.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:23.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:23.455 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:23.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:23.455 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:23.086+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:23.455 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:23.086+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:23.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:23.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:23.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:23.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:23.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:23.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:23.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:23.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:23.712+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:24.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:24.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:24.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:24.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:24.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:24.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:24.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:24.117+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:24.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:24.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:24.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:24.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:24.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:24.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:24.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:24.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:24.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:25.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:25.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:25.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:25.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:25.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:25.162+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:25.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:25.162+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:25.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:25.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:25.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:26.100 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:25.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:26.100 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:25.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:26.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:25.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:26.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:26.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:26.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:26.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:26.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:26.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:26.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:26.148+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:26.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:26.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:26.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:26.447+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:26.695+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:26.695+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:26.695+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:27.174 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:26.804+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41627 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:27.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:27.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:27.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:27.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:27.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:27.173+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:27.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:27.173+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:27.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:27.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:27.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:27.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:27.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:27.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:27.653+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:28.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:28.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:28.418+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:28.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:28.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:28.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:28.209+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:28.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:28.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:28.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:28.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:29.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:29.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:29.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:29.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:29.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:29.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:29.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:29.181+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:29.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:29.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:29.448+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:29.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:29.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:29.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:29.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:29.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:29.666+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:30.666 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:30.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:30.666 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:30.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:30.666 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:30.401+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:30.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:30.228+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:30.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:30.228+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:30.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:30.228+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:30.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:30.228+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:30.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:30.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:30.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:30.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:30.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:30.665+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:31.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:31.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:31.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:31.181+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:31.696 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:31.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:31.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:31.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:31.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:31.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:31.697 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:31.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:31.697 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:31.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:31.697 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:31.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:31.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:31.811+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41699 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:32.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:32.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:32.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:32.217+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:32.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:32.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:32.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:32.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:32.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:32.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:32.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:32.736+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:33.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:33.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:33.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:33.457+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:33.210+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:33.210+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:33.210+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:33.210+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:34.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:33.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:34.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:33.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:34.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:33.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:34.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:34.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:34.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:34.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:34.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:34.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:34.221+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:34.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:34.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:34.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:35.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:35.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:35.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:35.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:35.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:35.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:35.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:35.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:35.264+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:35.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:35.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:35.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:36.501 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:36.271+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:36.502 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:36.271+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:36.502 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:36.271+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:36.502 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:36.271+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:36.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:36.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:36.763 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:36.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:36.763 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:36.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:37.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:36.817+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41773 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:36.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:36.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:37.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:36.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:37.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:37.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:37.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:37.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:37.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:37.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:37.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:37.240+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:37.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:37.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:37.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:38.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:38.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:38.434+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:38.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:38.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:38.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:38.258+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:38.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:38.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:38.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:39.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:39.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:39.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:39.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:39.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:39.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:39.286+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:39.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:39.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:39.782+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:40.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:40.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:40.397+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:40.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:40.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:40.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:40.265+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:40.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:40.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:40.772+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:41.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:41.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:41.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:41.279+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:41.279+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:41.279+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:41.279+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:42.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:41.823+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41845 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:41.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:41.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:41.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:42.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:42.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:42.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:42.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:42.270+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:42.270+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:42.270+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:42.270+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:42.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:42.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:42.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:43.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:43.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:43.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:43.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:43.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:43.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:43.286+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:43.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:43.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:43.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:44.532 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:44.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:44.532 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:44.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:44.533 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:44.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:44.533 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:44.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:44.533 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:44.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:44.533 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:44.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:44.533 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:44.242+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:45.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:44.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:45.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:44.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:45.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:44.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:45.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:45.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:45.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:45.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:45.193+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:45.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:45.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:45.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:45.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:45.794 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:45.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:45.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:45.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:45.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:46.518 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:46.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:46.518 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:46.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:46.518 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:46.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:46.518 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:46.232+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:46.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:46.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:46.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:46.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:46.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:46.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:46.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:46.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:46.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:46.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:46.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:46.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:47.195 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:46.830+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41919 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:47.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:47.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:47.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:47.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:47.193+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:47.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:47.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:47.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:47.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:47.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:47.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:48.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:48.150+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:48.150+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:48.150+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:48.151+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:48.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:48.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:48.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:48.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:48.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:48.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:49.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:48.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:49.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:48.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:49.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:48.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:49.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:49.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:49.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:49.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:49.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:49.163+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:49.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:49.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:49.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:49.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:49.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:49.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:50.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:49.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:50.181 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:49.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:50.181 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:49.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:50.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:50.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:50.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:50.178+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:50.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:50.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:50.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:50.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:50.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:50.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:51.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:50.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:51.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:50.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:51.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:50.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:51.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:51.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:51.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:51.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:51.195+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:51.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:51.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:51.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:51.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:51.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:51.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:52.166 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:51.837+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 41991 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:52.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:51.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:52.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:51.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:52.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:51.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:52.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:52.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:52.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:52.164+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:52.840 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:52.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:52.840 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:52.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:52.840 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:52.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:53.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:52.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:53.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:52.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:53.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:52.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:53.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:53.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:53.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:53.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:53.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:53.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:53.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:53.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:53.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:53.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:53.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:53.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:53.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:53.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:53.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:54.122+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:54.122+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:54.122+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:54.122+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:54.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:54.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:54.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:54.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:54.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:54.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:55.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:54.888+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:55.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:54.888+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:55.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:54.888+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:55.144+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:55.144+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:55.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:55.144+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:55.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:55.144+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:55.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:55.499+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:55.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:55.499+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:55.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:55.499+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:56.183 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:55.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:56.183 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:55.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:56.183 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:55.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:56.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:56.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:56.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:56.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:56.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:56.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:56.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:56.181+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:56.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:56.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:56.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:56.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:57.173 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:02:56.844+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42065 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:02:57.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:56.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:57.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:56.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:57.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:56.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:57.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:57.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:57.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:57.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:57.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:57.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:57.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:57.171+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:57.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:57.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:57.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:57.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:58.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:57.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:58.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:57.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:58.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:57.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:58.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:58.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:58.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:58.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:58.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:58.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:58.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:58.170+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:58.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:58.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:58.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:59.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:58.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:59.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:58.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:59.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:58.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:59.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:59.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:02:59.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:02:59.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:59.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:59.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:59.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:59.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:59.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:02:59.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:02:59.188+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:59.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:59.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:59.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:02:59.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:02:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:02:59.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:00.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:00.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:00.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:00.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:00.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:00.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:00.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:00.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:00.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:00.218+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:01.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:00.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:01.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:00.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:01.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:00.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:01.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:01.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:01.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:01.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:01.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:01.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:01.169+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:02.145 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:01.850+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42137 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:02.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:01.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:02.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:01.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:02.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:01.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:02.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:02.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:02.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:02.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:02.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:02.384+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:02.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:02.142+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:02.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:02.142+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:02.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:02.142+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:02.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:02.142+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:02.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:02.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:02.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:03.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:03.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:03.124+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:03.124+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:03.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:03.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:03.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:04.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:03.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:04.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:03.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:04.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:03.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:04.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:04.083+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:04.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:04.083+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:04.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:04.083+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:04.367 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:04.083+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:04.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:04.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:04.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:05.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:04.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:05.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:04.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:05.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:04.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:05.129+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:05.129+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:05.129+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:05.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:05.129+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:05.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:05.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:05.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:05.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:05.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:05.345+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:06.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:05.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:06.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:05.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:06.178 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:05.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:06.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:06.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:06.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:06.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:06.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:06.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:06.176+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:06.176+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:07.206 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:06.857+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42211 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:07.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:06.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:07.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:06.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:07.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:06.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:07.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:07.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:07.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:07.204+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:07.204+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:07.204+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:07.205+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:08.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:07.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:08.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:07.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:08.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:07.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:08.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:08.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:08.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:08.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:08.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:08.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:08.170+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:09.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:08.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:09.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:08.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:09.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:08.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:09.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:09.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:09.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:09.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:09.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:09.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:09.184+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:10.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:09.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:10.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:09.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:10.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:09.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:10.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:10.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:10.315+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:10.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:10.203+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:10.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:10.203+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:10.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:10.203+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:10.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:10.203+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:11.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:10.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:11.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:10.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:11.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:10.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:11.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:11.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:11.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:11.307+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:11.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:11.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:11.224+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:11.224+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:12.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:11.863+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42283 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:11.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:11.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:11.909+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:12.272+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:12.272+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:12.272+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:12.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:12.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:12.233+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:12.233+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:13.220 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:12.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:13.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:12.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:13.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:12.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:13.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:13.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:13.289+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:13.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:13.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:13.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:13.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:13.219+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:14.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:13.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:14.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:13.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:14.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:13.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:14.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:14.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:14.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:14.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:14.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:14.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:14.200+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:15.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:14.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:15.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:14.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:15.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:14.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:15.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:15.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:15.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:15.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:15.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:15.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:15.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:15.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:15.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:15.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:15.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:15.178+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:16.205 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:15.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:16.205 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:15.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:16.205 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:15.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:16.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:16.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:16.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:16.203+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:16.203+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:16.204+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:16.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:16.204+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:17.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:16.869+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42357 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:17.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:16.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:17.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:16.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:17.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:16.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:17.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:17.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:17.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:17.234+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:17.234+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:17.234+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:17.234+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:17.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:17.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:17.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:18.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:18.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:18.288+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:18.276+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:18.276+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:18.276+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:18.276+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:18.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:18.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:18.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:19.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:19.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:19.308+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:19.230+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:19.230+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:19.230+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:19.231+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:20.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:19.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:20.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:19.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:20.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:19.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:20.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:20.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:20.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:20.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:20.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:20.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:20.195+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:21.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:20.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:21.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:20.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:21.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:20.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:21.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:21.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:21.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:21.374+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:21.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:21.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:21.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:21.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:21.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:21.193+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:22.169 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:21.876+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42429 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:22.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:21.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:22.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:21.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:22.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:21.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:22.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:22.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:22.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:22.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:22.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:22.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:22.168+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:23.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:22.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:23.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:22.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:23.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:22.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:23.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:23.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:23.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:23.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:23.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:23.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:23.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:23.175+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:24.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:23.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:24.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:23.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:24.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:23.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:24.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:24.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:24.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:24.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:24.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:24.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:24.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:24.192+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:24.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:24.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:24.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:24.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:24.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:24.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:25.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:24.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:25.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:24.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:25.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:24.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:25.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:25.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:25.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:25.157+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:25.157+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:25.157+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:25.157+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:26.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:25.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:26.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:25.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:26.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:25.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:26.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:26.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:26.347+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:26.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:26.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:26.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:26.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:27.156 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:26.883+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42502 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:27.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:26.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:27.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:26.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:27.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:26.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:27.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:27.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:27.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:27.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:27.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:27.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:27.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:28.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:27.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:28.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:27.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:28.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:27.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:28.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:28.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:28.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:28.197+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:28.197+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:28.197+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:28.197+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:29.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:28.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:29.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:28.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:29.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:28.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:29.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:29.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:29.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:29.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:29.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:29.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:29.172+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:29.172+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:30.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:29.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:30.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:29.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:30.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:29.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:30.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:30.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:30.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:30.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:30.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:30.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:30.175+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:31.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:30.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:31.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:30.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:31.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:30.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:31.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:31.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:31.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:31.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:31.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:31.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:31.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:31.191+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:32.179 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:31.889+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42575 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:32.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:31.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:32.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:31.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:32.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:31.917+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:32.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:32.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:32.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:32.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:32.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:32.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:32.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:32.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:32.177+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:32.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:32.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:32.947+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:33.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:33.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:33.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:33.190+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:33.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:33.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:33.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:33.314+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:34.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:33.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:34.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:33.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:34.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:33.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:34.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:34.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:34.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:34.276+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:34.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:34.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:34.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:34.184+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:35.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:34.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:35.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:34.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:35.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:34.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:35.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:35.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:35.302+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:35.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:35.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:35.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:35.168+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:36.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:35.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:36.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:35.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:36.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:35.919+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:36.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:36.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:36.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:36.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:36.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:36.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:36.180+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:37.021 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:36.895+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42649 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:37.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:36.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:37.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:36.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:37.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:36.946+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:37.287 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:37.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:37.287 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:37.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:37.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:37.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:37.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:37.168+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:37.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:37.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:37.285+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:37.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:37.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:37.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:38.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:38.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:38.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:38.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:38.156+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:38.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:38.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:38.263+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:38.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:38.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:38.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:39.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:39.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:39.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:39.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:39.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:39.237+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:39.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:39.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:39.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:39.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:39.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:39.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:39.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:39.184+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:40.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:40.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:40.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:40.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:40.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:40.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:40.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:40.035+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:40.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:40.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:40.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:40.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:40.146+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:41.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:41.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:41.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:41.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:41.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:41.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:41.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:41.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:41.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:41.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:41.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:41.174+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:41.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:41.174+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:42.199 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:41.903+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42722 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:42.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:42.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:42.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:42.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:42.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:42.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:42.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:42.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:42.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:42.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:42.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:42.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:42.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:42.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:42.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:42.197+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:43.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:43.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:43.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:43.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:43.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:43.193+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:43.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:43.150+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:43.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:43.150+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:43.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:43.150+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:43.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:43.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:43.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:43.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:43.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:44.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:44.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:44.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:44.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:44.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:44.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:44.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:44.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:44.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:44.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:44.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:44.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:44.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:44.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:44.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:44.236+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:45.085+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:45.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:45.085+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:45.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:45.085+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:45.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:45.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:45.025+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:45.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:45.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:45.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:45.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:45.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:45.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:45.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:45.214+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:46.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:46.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:46.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:46.065+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:46.065+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:46.065+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:46.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:46.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:46.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:46.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:46.200+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:47.214 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:46.908+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42795 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:47.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:47.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:47.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:47.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:47.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:47.055+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:47.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:47.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:47.215 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:47.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:47.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:47.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:47.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:47.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:47.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:47.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:48.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:48.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:48.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:48.199+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:48.199+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:48.199+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:48.199+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:48.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:48.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:48.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:48.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:48.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:48.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:49.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:49.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:49.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:49.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:49.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:49.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:49.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:49.194+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:49.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:49.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:49.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:49.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:49.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:49.074+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:50.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:50.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:50.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:50.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:50.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:50.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:50.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:50.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:50.155+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:50.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:50.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:50.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:51.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:51.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:51.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:51.140+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:51.140+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:51.140+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:51.140+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:51.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:50.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:51.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:50.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:51.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:50.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:52.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:51.915+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42868 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:52.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:52.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:52.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:52.164+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:52.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:52.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:52.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:52.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:52.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:52.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:52.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:52.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:52.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:52.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:52.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:52.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:53.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:53.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:53.063+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:53.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:53.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:53.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:53.188+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:53.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:53.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:53.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:54.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:54.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:54.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:54.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:54.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:54.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:54.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:54.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:54.169+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:54.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:54.169+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:55.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:55.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:55.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:55.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:55.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:55.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:55.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:55.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:55.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:55.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:55.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:55.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:55.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:55.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:56.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:56.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:56.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:56.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:56.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:56.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:56.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:56.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:56.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:56.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:56.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:56.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:56.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:56.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:56.190+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:56.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:56.190+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:57.082 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:03:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:03:56.923+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 42941 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:03:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:57.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:57.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:57.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:57.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:57.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:57.081+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:57.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:57.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:57.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:57.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:57.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:57.171+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:57.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:57.172+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:58.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:58.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:58.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:58.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:58.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:58.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:58.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:58.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:58.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:58.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:58.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:58.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:58.163+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:59.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:59.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:03:59.086+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:03:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:59.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:59.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:03:59.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:03:59.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:59.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:59.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:59.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:59.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:59.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:03:59.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:03:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:03:59.206+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:00.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:00.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:00.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:00.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:00.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:00.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:00.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:00.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:00.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:00.247+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:00.247+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:00.247+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:00.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:00.247+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:01.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:01.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:01.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:01.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:01.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:01.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:01.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:01.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:01.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:01.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:01.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:01.233+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:02.191 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:01.931+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43014 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:02.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:02.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:02.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:02.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:02.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:02.104+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:02.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:02.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:02.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:02.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:02.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:02.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:02.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:02.189+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:02.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:02.189+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:02.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:02.189+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:02.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:02.190+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:03.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:03.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:03.176+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:03.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:03.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:03.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:03.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:03.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:03.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:03.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:03.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:03.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:03.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:03.186+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:04.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:04.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:04.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:04.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:04.405 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:04.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:04.405 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:04.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:04.405 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:04.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:04.405 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:04.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:04.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:04.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:04.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:04.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:04.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:04.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:04.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:04.164+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:05.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:05.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:05.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:05.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:05.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:05.097+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:05.097+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:05.097+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:05.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:05.145+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:05.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:05.145+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:05.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:05.145+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:06.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:06.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:06.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:06.178+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:06.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:06.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:06.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:06.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:06.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:06.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:06.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:06.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:06.195+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:06.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:06.196+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:07.094 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:06.937+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43087 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:07.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:07.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:07.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:07.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:07.212+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:07.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:07.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:07.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:07.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:07.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:07.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:07.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:07.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:07.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:07.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:07.481 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:07.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:08.155+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:08.155+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:08.155+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:08.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:08.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:08.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:08.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:08.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:08.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:08.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:08.255+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:08.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:08.255+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:08.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:08.255+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:08.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:08.255+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:09.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:09.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:09.169+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:09.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:09.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:09.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:09.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:09.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:09.057+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:09.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:09.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:09.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:09.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:09.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:09.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:09.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:09.227+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:10.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:10.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:10.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:10.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:10.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:10.103+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:10.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:10.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:10.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:10.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:10.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:10.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:10.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:10.214+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:11.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:11.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:11.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:11.162+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:11.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:11.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:11.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:11.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:11.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:11.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:11.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:11.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:11.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:11.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:11.219+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:12.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:11.945+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43160 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:12.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:12.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:12.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:12.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:12.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:12.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:12.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:12.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:12.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:12.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:12.262+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:12.262+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:12.262+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:12.262+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:13.295 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:13.155+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:13.296 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:13.155+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:13.296 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:13.155+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:13.296 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:13.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:13.296 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:13.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:13.296 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:13.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:13.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:13.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:13.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:13.294+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:14.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:13.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:14.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:13.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:14.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:13.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:14.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:14.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:14.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:14.326+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:14.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:14.326+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:14.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:14.326+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:14.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:14.326+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:15.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:15.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:15.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:15.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:15.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:14.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:14.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:14.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:15.368+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:15.368+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:15.368+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:15.369+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:16.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:16.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:16.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:16.104+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:15.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:15.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:15.963+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:16.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:16.383+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:16.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:16.383+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:16.692 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:16.383+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:16.693 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:16.383+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:16.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:16.933+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:16.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:16.933+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:16.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:16.933+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:17.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:16.952+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43234 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:17.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:17.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:17.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:17.413+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:17.413+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:17.413+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:17.413+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:18.122+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:18.122+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:18.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:18.122+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:17.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:17.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:17.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:18.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:18.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:18.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:18.453+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:19.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:19.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:19.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:18.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:18.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:18.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:19.471+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:19.471+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:19.471+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:19.471+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:20.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:20.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:20.052+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:20.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:19.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:20.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:19.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:20.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:19.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:20.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:20.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:20.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:20.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:20.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:20.484+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:20.484+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:21.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:21.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:21.018+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:21.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:20.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:21.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:20.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:21.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:20.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:21.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:21.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:21.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:21.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:21.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:21.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:21.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:21.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:22.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:21.958+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43307 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:21.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:21.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:22.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:21.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:22.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:22.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:22.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:22.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:22.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:22.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:22.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:22.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:22.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:22.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:22.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:22.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:22.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:22.482+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:22.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:22.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:22.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:23.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:23.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:23.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:23.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:23.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:23.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:23.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:23.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:23.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:23.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:24.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:23.802+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:24.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:23.802+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:24.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:23.802+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:24.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:24.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:24.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:24.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:24.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:24.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:24.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:24.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:24.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:24.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:24.491+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:25.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:24.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:25.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:24.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:25.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:24.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:25.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:25.020+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:25.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:25.020+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:25.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:25.020+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:25.465+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:25.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:25.466+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:25.466+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:26.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:25.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:26.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:25.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:26.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:25.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:26.443 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:26.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:26.443 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:26.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:26.444 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:26.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:26.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:26.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:26.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:26.441+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:27.088 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:26.964+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43380 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:27.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:26.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:27.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:26.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:27.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:26.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:27.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:27.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:27.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:27.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:27.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:27.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:27.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:27.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:27.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:27.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:27.467+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:27.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:27.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:27.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:28.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:28.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:28.046+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:28.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:28.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:28.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:28.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:28.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:28.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:28.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:28.492+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:28.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:28.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:28.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:29.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:29.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:29.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:29.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:29.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:29.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:29.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:29.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:29.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:29.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:29.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:29.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:29.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:29.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:30.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:30.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:30.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:29.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:29.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:29.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:30.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:30.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:30.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:30.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:31.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:31.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:31.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:30.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:30.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:30.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:31.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:31.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:31.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:31.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:31.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:31.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:31.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:31.505+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:32.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:31.971+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43453 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:32.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:32.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:32.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:31.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:31.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:31.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:32.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:32.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:32.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:32.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:32.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:32.492+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:32.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:32.492+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:32.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:32.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:32.971+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:32.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:32.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:32.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:33.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:33.445+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:33.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:33.445+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:33.445+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:33.445+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:34.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:33.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:34.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:33.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:34.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:33.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:33.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:33.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:33.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:34.412+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:34.412+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:34.412+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:34.412+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:35.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:35.004+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:35.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:35.004+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:35.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:35.004+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:34.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:34.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:34.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:35.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:35.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:35.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:35.454+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:36.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:36.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:36.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:35.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:35.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:35.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:36.417+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:36.417+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:36.417+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:36.417+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:37.105 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:36.977+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43526 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:37.106 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:36.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:37.106 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:36.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:37.106 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:36.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:37.107 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:36.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:37.107 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:36.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:37.107 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:36.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:37.394+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:37.394+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:37.394+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:37.394+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:38.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:38.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:38.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:38.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:37.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:37.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:37.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:38.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:38.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:38.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:38.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:38.350+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:39.323 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:39.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:39.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:39.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:39.324 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:39.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:39.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:39.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:39.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:39.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:39.324 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:39.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:39.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:39.322+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:39.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:39.322+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:39.322+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:39.322+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:39.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:39.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:39.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:39.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:39.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:39.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:40.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:40.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:40.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:40.350+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:40.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:40.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:41.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:40.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:40.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:40.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:40.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:41.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:41.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:41.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:41.378+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:42.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:41.985+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43599 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:41.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:41.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:41.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:41.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:41.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:41.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:42.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:42.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:42.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:42.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:42.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:42.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:42.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:42.391+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:42.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:42.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:43.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:42.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:42.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:42.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:42.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:43.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:43.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:43.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:43.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:43.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:43.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:43.391+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:44.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:43.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:44.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:43.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:44.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:43.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:43.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:43.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:44.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:43.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:44.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:44.405+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:44.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:44.405+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:44.405+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:44.405+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:44.948+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:44.948+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:44.948+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:44.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:44.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:45.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:44.995+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:45.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:45.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:45.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:45.425+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:45.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:45.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:46.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:45.998+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:45.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:45.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:46.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:45.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:46.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:46.450+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:46.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:46.450+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:46.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:46.450+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:46.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:46.451+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:47.118 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:46.993+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43672 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:47.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:46.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:47.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:46.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:47.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:46.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:47.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:47.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:47.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:47.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:47.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:47.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:47.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:47.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:47.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:47.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:47.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:47.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:47.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:47.480+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:48.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:47.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:48.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:47.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:47.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:47.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:47.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:47.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:48.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:48.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:48.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:48.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:48.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:48.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:48.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:48.509+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:48.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:48.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:48.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:48.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:48.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:48.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:49.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:49.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:49.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:49.464+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:49.464+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:49.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:49.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:49.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:49.891+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:49.891+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:49.891+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:50.424+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:50.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:50.424+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:50.424+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:50.424+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:50.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:50.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:50.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:50.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:50.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:04:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:04:50.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.963824+0000 (oldest deadline 2024-02-19T19:54:40.963824+0000) 2024-02-19T20:04:51.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:51.402+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:51.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:51.402+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:51.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:51.402+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:51.732 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:51.402+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:52.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:52.000+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43745 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:51.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:51.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:51.879+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:52.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:52.419+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:52.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:52.419+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:52.419+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:52.419+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:52.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:52.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:04:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:04:52.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:20.623401+0000 (oldest deadline 2024-02-19T19:54:40.623401+0000) 2024-02-19T20:04:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:53.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:53.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:53.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:53.458+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:54.471+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:54.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:54.471+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:54.471+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:54.471+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:55.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:55.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:55.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:55.447+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:56.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:56.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:56.439+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:56.439+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:57.133 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:04:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:04:57.006+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43818 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:04:57.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:57.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:57.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:57.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:57.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:57.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:57.475+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:58.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:58.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:58.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:58.524+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:59.507+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:59.507+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:59.507+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:04:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:04:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:04:59.507+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:00.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:00.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:00.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:00.531+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:01.483+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:01.483+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:01.483+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:01.483+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:02.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:02.014+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43891 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:02.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:02.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:02.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:02.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:03.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:03.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:03.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:54:30.720789+0000 (oldest deadline 2024-02-19T19:54:50.720789+0000) 2024-02-19T20:05:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:03.546+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:04.506+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:05.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:05.512+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:06.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:06.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:07.149 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:07.021+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 43964 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:07.600+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:08.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:08.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:09.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:09.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:10.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:10.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:11.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:11.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:12.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:12.027+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44037 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:12.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:12.575+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:12.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:12.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:12.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:13.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:13.609+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:13.850+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:13.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:13.850+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:13.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:13.850+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:13.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:13.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:13.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:14.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:14.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:14.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:14.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:14.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:14.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:14.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:14.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:15.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:15.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:15.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:15.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:15.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:15.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:15.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:15.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:15.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:15.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:16.708+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:16.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:16.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:16.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:16.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:16.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:16.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:16.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:16.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:16.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:17.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:17.034+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44110 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:17.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:17.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:17.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:17.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:17.776+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:17.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:17.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:17.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:17.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:17.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:17.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:18.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:18.770+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:18.770+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:18.770+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:18.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:18.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:18.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:18.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:19.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:19.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:19.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:19.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:19.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:19.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:19.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:19.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:20.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:20.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:20.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:20.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:20.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:20.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:20.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:20.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:20.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:20.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:20.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:21.674+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:21.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:21.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:21.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:21.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:21.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:21.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:22.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:22.041+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44183 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:22.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:22.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:22.851+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:22.851+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:22.851+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:22.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:22.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:22.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:22.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:24.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:23.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:23.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:23.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:23.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:23.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:23.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:23.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:24.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:24.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:24.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:24.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:24.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:24.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:24.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:25.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:25.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:25.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:25.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:25.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:25.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:25.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:25.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:25.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:25.816+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:26.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:26.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:26.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:26.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:26.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:26.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:26.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:26.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:26.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:27.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:26.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:27.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:27.047+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44256 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:27.788+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:27.788+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:27.788+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:27.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:27.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:27.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:27.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:27.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:27.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:27.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:28.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:28.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:28.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:28.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:28.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:28.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:28.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:28.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:28.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:28.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:28.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:28.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:28.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:28.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:28.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:28.855+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:29.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:29.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:29.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:29.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:29.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:29.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:29.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:29.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:29.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:29.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:29.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:29.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:29.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:29.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:29.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:30.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:30.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:30.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:30.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:30.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:30.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:30.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:30.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:30.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:30.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:30.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:30.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:30.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:30.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:30.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:30.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:30.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:30.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:30.954 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:30.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:31.849+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:31.849+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:31.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:31.849+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:31.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:31.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:31.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:31.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:31.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:31.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:31.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:31.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:32.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:32.054+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44329 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:32.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:32.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:32.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:32.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:32.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:32.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:32.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:32.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:32.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:32.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:32.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:32.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:32.692+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:33.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:33.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:33.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:33.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:33.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:33.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:33.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:33.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:33.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:33.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:33.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:34.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:34.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:34.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:34.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:34.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:34.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:34.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:34.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:34.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:34.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:35.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:35.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:35.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:35.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:35.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:35.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:35.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:35.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:35.865+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:35.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:35.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:35.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:35.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:36.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:36.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:36.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:36.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:37.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:36.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:37.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:36.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:37.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:36.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:37.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:36.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:37.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:36.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:37.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:36.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:37.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:37.062+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44402 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:37.912 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:37.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:37.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:37.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:37.913 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:37.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:37.913 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:37.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:37.913 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:37.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:37.913 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:37.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:37.913 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:37.658+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:37.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:37.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:38.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:37.911+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:38.903 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:38.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:38.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:38.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:38.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:38.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:38.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:38.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:38.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:38.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:38.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:38.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:38.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:38.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:38.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:38.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:39.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:38.902+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:39.870+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:39.870+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:39.870+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:39.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:39.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:39.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:39.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:39.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:39.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:39.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:39.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:39.681+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:39.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:40.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:40.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:40.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:40.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:40.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:40.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:40.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:40.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:40.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:40.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:41.888+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:41.888+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:41.888+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:41.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:41.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:41.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:41.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:41.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:41.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:41.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:41.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:41.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:41.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:42.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:42.069+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44475 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:42.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:42.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:42.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:42.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:42.940 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:42.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:42.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:42.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:42.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:42.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:42.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:42.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:42.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:42.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:42.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:42.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:43.231 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:42.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:43.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:43.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:43.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:43.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:43.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:43.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:43.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:43.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:44.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:43.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:44.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:43.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:44.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:44.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:44.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:44.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:44.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:44.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:44.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:44.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:44.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:44.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:44.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:44.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:45.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:44.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:45.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:45.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:45.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:45.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:45.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:45.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:45.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:45.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:45.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:45.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:45.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:46.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:46.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:46.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:46.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:46.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:46.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:46.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:46.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:47.200 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:47.076+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44551 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:47.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:46.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:47.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:46.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:47.201 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:46.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:47.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:47.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:47.202 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:47.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:47.202 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:47.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:47.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:47.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:47.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:47.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:47.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:47.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:47.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:47.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:48.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:48.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:48.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:47.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:47.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:47.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:48.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:48.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:48.889 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:48.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:48.889 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:48.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:48.889 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:48.622+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:48.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:48.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:48.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:48.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:48.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:48.997+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:49.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:49.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:49.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:49.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:49.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:49.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:49.860 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:49.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:49.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:49.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:49.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:50.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:50.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:50.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:50.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:50.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:50.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:50.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:50.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:50.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:50.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:50.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:51.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:51.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:51.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:51.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:51.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:51.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:51.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:51.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:51.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:51.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:51.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:51.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:52.208 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:52.083+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44624 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:52.208 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:51.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:52.209 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:51.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:52.209 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:51.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:52.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:52.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:52.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:52.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:52.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:52.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:52.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:52.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:52.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:52.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:52.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:52.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:52.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:53.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:53.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:53.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:53.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:53.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:53.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:53.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:53.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:53.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:53.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:53.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:54.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:54.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:54.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:54.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:54.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:54.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:54.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:54.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:54.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:54.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:54.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:55.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:55.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:55.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:55.010+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:55.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:55.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:55.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:55.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:55.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:55.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:55.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:55.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:55.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:56.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:55.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:56.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:55.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:56.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:56.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:56.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:56.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:56.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:56.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:56.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:56.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:56.617+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:56.964 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:56.617+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:57.216 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:05:57.089+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44697 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:05:57.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:57.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:57.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:57.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:57.216 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:57.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:57.216 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:56.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:57.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:56.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:57.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:56.961+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:57.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:57.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:57.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:57.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:57.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:57.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:57.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:57.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:58.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:58.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:58.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:58.023+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:57.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:57.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:57.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:58.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:58.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:58.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:58.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:59.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:59.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:05:59.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:05:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:59.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:59.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:59.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:05:59.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:59.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:59.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:59.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:59.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:59.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:05:59.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:05:59.589+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:00.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:00.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:00.050+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:59.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:59.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:05:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:05:59.974+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:00.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:00.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:00.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:00.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:00.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:00.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:00.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:00.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:01.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:01.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:01.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:00.970+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:01.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:01.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:01.926 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:01.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:01.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:01.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:01.927 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:01.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:02.222 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:02.095+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44770 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:02.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:01.924+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:02.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:01.924+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:02.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:01.924+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:02.223 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:02.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:02.223 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:02.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:02.223 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:02.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:02.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:02.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:02.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:02.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:02.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:02.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:02.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:02.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:03.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:03.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:03.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:03.033+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:02.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:02.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:02.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:03.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:03.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:03.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:03.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:03.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:03.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:03.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:04.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:04.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:04.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:04.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:04.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:04.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:04.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:04.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:04.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:04.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:04.630+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:05.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:05.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:05.072+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:04.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:04.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:04.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:05.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:05.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:05.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:05.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:05.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:05.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:05.914 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:05.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:06.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:06.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:06.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:06.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:06.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:06.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:06.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:05.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:06.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:05.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:06.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:05.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:06.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:06.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:06.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:06.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:06.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:06.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:06.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:06.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:07.227 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:07.103+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44843 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:07.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:07.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:07.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:07.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:06.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:06.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:06.901+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:07.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:07.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:07.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:07.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:07.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:07.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:07.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:07.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:07.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:07.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:07.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:08.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:08.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:08.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:08.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:08.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:08.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:08.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:08.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:08.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:08.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:09.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:09.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:09.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:09.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:09.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:09.578+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:09.578+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:10.006+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:10.006+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:10.006+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:10.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:10.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:10.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:10.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:10.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:10.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:10.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:11.287 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:11.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:11.288 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:11.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:11.288 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:11.001+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:11.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:11.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:11.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:11.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:11.961 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:11.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:11.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:11.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:11.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:11.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:11.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:11.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:11.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:11.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:12.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:12.109+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44916 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:12.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:11.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:12.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:11.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:12.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:11.959+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:12.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:12.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:12.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:12.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:12.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:12.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:12.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:12.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:12.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:12.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:12.967+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:12.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:12.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:12.825+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:13.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:13.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:13.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:13.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:13.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:13.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:13.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:13.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:14.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:14.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:14.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:14.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:14.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:14.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:13.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:14.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:13.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:14.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:13.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:14.820 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:14.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:14.821 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:14.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:14.821 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:14.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:14.821 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:14.498+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:15.029+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:15.029+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:15.029+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:14.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:14.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:15.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:14.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:15.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:15.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:15.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:15.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:15.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:15.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:15.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:15.543+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:16.329 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:16.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:16.330 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:16.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:16.330 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:16.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:16.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:16.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:16.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:16.508+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:17.027 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:16.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:17.027 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:16.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:17.028 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:16.761+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:17.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:17.115+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 44989 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:17.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:17.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:17.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:17.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:17.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:17.025+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:17.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:17.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:17.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:17.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:17.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:17.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:17.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:18.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:18.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:18.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:18.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:18.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:18.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:18.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:18.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:18.564+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:18.564+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:19.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:19.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:19.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:19.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:19.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:19.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:19.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:19.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:19.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:19.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:19.589+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:19.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:19.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:19.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:20.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:20.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:20.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:20.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:20.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:20.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:20.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:20.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:20.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:20.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:21.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:20.831+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:21.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:21.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:21.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:21.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:21.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:21.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:21.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:22.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:22.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:22.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:22.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:22.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:22.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:22.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:21.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:22.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:21.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:22.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:21.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:22.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:22.122+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45062 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:22.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:22.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:22.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:22.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:22.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:22.575+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:22.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:22.575+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:23.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:23.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:23.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:23.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:23.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:23.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:22.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:22.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:22.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:23.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:23.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:23.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:23.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:23.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:23.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:23.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:23.545+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:23.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:23.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:23.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:24.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:24.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:24.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:24.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:24.835 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:24.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:24.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:24.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:24.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:24.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:24.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:24.526+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:25.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:25.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:25.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:25.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:25.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:24.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:24.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:24.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:25.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:25.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:25.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:25.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:25.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:25.514+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:25.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:25.514+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:26.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:26.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:26.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:26.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:26.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:26.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:26.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:25.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:26.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:25.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:26.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:25.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:26.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:26.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:26.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:26.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:27.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:26.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:27.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:26.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:27.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:26.850+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:27.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:27.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:27.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:27.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:27.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:27.057+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:27.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:27.130+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45135 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:27.862 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:27.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:27.862 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:27.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:27.863 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:27.509+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:27.863 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:27.509+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:28.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:28.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:28.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:28.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:28.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:28.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:28.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:27.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:28.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:27.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:28.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:27.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:28.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:28.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:28.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:28.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:28.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:28.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:28.868 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:28.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:28.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:28.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:28.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:28.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:28.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:29.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:28.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:29.907 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:29.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:29.907 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:29.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:29.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:29.531+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:29.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:29.531+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:29.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:29.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:30.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:29.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:29.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:29.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:29.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:30.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:30.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:30.894 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:30.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:30.894 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:30.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:30.894 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:30.542+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:30.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:30.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:30.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:30.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:30.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:30.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:31.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:31.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:31.544+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:31.544+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:32.158 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:32.137+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45208 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:32.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:31.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:32.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:31.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:32.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:31.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:32.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:31.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:32.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:31.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:32.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:31.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:32.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:32.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:32.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:32.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:32.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:32.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:32.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:32.511+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:33.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:33.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:33.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:32.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:32.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:32.880+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:33.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:33.507+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:33.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:33.507+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:33.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:33.507+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:33.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:33.507+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:34.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:33.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:34.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:33.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:33.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:33.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:33.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:34.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:33.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:34.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:34.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:34.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:34.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:34.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:34.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:34.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:34.542+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:35.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:34.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:35.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:34.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:35.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:34.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:34.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:34.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:35.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:34.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:35.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:35.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:35.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:35.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:35.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:35.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:35.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:35.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:35.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:35.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:36.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:35.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:35.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:35.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:36.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:35.923+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:36.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:36.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:36.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:36.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:37.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:36.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:37.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:36.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:37.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:36.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:37.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:36.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:37.147 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:36.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:37.147 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:36.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:37.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:37.144+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45281 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:37.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:37.495+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:37.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:37.495+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:37.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:37.495+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:37.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:37.495+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:37.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:38.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:37.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:38.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:37.868+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:37.931+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:37.931+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:37.931+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:38.822 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:38.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:38.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:38.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:38.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:38.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:38.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:38.521+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:38.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:38.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:39.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:38.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:38.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:38.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:38.905+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:39.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:39.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:39.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:39.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:39.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:39.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:39.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:39.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:39.855+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:39.855+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:39.855+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:39.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:39.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:39.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:40.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:40.535+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:40.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:40.535+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:40.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:40.535+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:40.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:40.535+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:40.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:40.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:41.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:40.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:40.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:40.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:40.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:41.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:41.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:41.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:41.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:41.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:41.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:41.819 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:41.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:42.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:41.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:42.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:41.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:42.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:41.818+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:42.154 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:41.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:42.154 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:41.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:42.154 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:41.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:42.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:42.151+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45354 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:42.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:42.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:42.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:42.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:42.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:42.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:42.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:42.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:42.855+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:42.855+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:43.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:42.855+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:42.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:42.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:42.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:43.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:43.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:43.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:43.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:43.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:43.546+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:43.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:43.546+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:44.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:43.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:44.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:43.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:44.149 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:43.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:44.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:43.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:44.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:43.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:44.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:43.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:44.797 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:44.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:44.797 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:44.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:44.797 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:44.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:44.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:44.502+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:44.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:44.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:44.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:44.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:44.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:45.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:44.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:45.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:45.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:45.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:45.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:45.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:45.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:45.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:45.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:45.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:45.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:45.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:45.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:45.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:45.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:45.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:45.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:45.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:46.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:46.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:46.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:46.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:46.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:46.562+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:46.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:46.562+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:46.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:46.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:46.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:46.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:46.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:46.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:47.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:46.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:47.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:46.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:47.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:46.873+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:47.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:47.157+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45428 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:47.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:47.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:47.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:47.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:47.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:47.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:47.520+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:48.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:47.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:47.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:47.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:48.927 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:48.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:48.927 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:48.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:48.928 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:48.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:48.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:48.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:48.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:48.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:48.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:48.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:48.928 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:48.535+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:48.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:48.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:48.925+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:49.962 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:49.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:49.962 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:49.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:49.962 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:49.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:49.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:49.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:49.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:49.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:49.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:49.557+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:49.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:49.557+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:49.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:49.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:50.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:49.960+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:50.919 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:50.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:50.919 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:50.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:50.919 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:50.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:50.919 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:50.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:50.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:50.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:50.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:50.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:50.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:50.529+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:50.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:50.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:50.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:51.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:51.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:51.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:51.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:51.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:51.675+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:51.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:51.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:51.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:51.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:51.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:51.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:51.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:51.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:52.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:52.165+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45498 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:51.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:51.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:51.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:52.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:52.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:52.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:52.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:52.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:52.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:52.679 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:52.504+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:52.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:52.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:52.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:52.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:52.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:52.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:52.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:52.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:52.966+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:53.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:53.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:53.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:53.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:53.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:53.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:53.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:53.517+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:53.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:53.517+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:53.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:53.517+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:53.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:53.517+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:53.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:53.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:53.967+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:54.922 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:54.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:54.922 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:54.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:54.923 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:54.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:54.923 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:54.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:54.923 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:54.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:54.923 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:54.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:54.923 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:54.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:54.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:54.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:55.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:54.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:55.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:55.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:55.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:55.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:55.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:55.549+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:55.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:55.549+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:55.924 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:55.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:55.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:55.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:55.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:55.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:55.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:55.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:56.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:55.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:56.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:56.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:56.501+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:56.501+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:57.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:56.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:57.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:56.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:57.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:56.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:57.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:56.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:57.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:56.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:57.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:56.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:57.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:06:57.171+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45565 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:06:57.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:57.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:57.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:57.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:57.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:57.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:57.784 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:57.487+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:57.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:57.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:57.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:57.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:57.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:57.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:58.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:58.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:58.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:58.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:58.766 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:58.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:58.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:58.496+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:58.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:58.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:58.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:06:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:58.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:58.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:58.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:06:59.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:59.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:59.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:59.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:06:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:06:59.474+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:59.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:59.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:06:59.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:59.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:59.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:06:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:06:59.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:00.902 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:00.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:00.902 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:00.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:00.902 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:00.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:00.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:00.499+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:00.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:00.499+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:00.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:00.499+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:00.903 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:00.500+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:00.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:00.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:00.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:01.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:01.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:01.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:01.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:01.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:01.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:01.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:01.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:01.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:01.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:01.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:01.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:01.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:01.527+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:02.190 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:02.178+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45632 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:02.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:01.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:02.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:01.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:02.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:01.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:02.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:02.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:02.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:02.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:02.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:02.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:02.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:02.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:02.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:02.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:02.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:02.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:02.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:02.510+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:02.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:02.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:02.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:03.865 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:03.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:03.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:03.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:03.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:03.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:03.866 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:03.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:03.866 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:03.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:03.866 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:03.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:03.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:03.548+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:03.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:03.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:03.863+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:04.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:04.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:04.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:04.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:04.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:04.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:04.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:04.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:04.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:04.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:04.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:04.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:04.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:05.828 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:05.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:05.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:05.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:05.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:05.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:05.829 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:05.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:05.829 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:05.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:05.829 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:05.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:05.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:05.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:05.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:05.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:06.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:05.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:06.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:06.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:06.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:06.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:06.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:06.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:06.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:06.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:06.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:06.565+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:06.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:06.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:06.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:06.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:06.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:06.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:07.309 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:07.185+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45696 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:07.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:07.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:07.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:07.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:07.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:07.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:07.584+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:07.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:07.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:07.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:07.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:07.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:08.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:08.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:08.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:08.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:08.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:08.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:08.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:08.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:08.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:08.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:08.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:08.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:09.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:09.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:09.670+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:09.714+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:09.714+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:09.714+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:09.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:09.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:09.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:09.631+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:10.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:10.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:10.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:10.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:10.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:10.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:10.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:10.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:10.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:10.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:11.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:11.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:11.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:11.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:11.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:11.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:11.612+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:11.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:11.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:11.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:11.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:11.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:11.726+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:12.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:12.191+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45759 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:12.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:12.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:12.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:12.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:12.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:12.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:12.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:12.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:12.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:12.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:12.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:12.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:12.694+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:13.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:13.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:13.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:13.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:13.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:13.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:13.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:13.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:13.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:13.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:14.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:14.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:14.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:14.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:14.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:14.674+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:14.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:14.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:14.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:14.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:15.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:15.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:15.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:15.717+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:15.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:15.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:15.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:15.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:15.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:15.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:15.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:16.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:16.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:16.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:16.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:16.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:16.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:16.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:16.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:16.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:16.671+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:16.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:16.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:16.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:16.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:16.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:16.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:16.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:16.616+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:17.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:17.199+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45830 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:17.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:17.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:17.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:17.713+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:17.714+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:17.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:17.714+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:17.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:17.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:17.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:17.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:17.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:17.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:17.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:18.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:18.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:18.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:18.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:18.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:18.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:18.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:18.725+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:18.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:18.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:18.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:18.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:18.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:18.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:18.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:18.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:19.750+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:19.750+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:19.750+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:19.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:19.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:19.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:19.623+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:19.623+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:19.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:19.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:19.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:19.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:20.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:20.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:20.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:20.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:20.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:20.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:20.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:20.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:20.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:20.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:21.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:21.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:21.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:21.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:21.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:21.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:21.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:21.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:21.660+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:21.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:22.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:22.206+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45899 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:22.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:22.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:22.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:22.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:22.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:22.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:22.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:22.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:22.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:22.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:22.799+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:23.848 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:23.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:23.848 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:23.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:23.848 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:23.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:23.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:23.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:23.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:23.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:23.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:23.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:23.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:23.573+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:23.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:23.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:24.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:23.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:24.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:24.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:24.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:24.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:24.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:24.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:24.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:24.600+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:24.600+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:24.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:24.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:24.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:24.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:24.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:24.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:25.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:25.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:25.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:25.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:25.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:25.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:25.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:25.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:25.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:25.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:25.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:26.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:26.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:26.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:26.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:26.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:26.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:26.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:26.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:26.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:26.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:26.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:26.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:26.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:26.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:26.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:26.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:26.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:26.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:27.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:27.213+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 45972 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:27.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:27.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:27.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:27.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:27.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:27.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:27.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:27.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:27.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:27.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:27.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:28.895 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:28.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:28.895 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:28.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:28.895 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:28.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:28.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:28.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:28.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:28.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:28.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:28.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:28.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:28.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:28.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:28.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:28.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:29.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:29.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:29.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:29.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:29.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:29.625+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:29.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:29.625+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:29.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:29.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:29.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:29.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:29.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:29.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:29.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:29.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:29.939+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:30.924 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:30.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:30.924 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:30.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:30.924 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:30.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:30.924 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:30.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:30.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:30.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:30.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:30.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:30.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:30.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:30.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:30.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:30.922+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:31.908 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:31.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:31.908 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:31.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:31.908 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:31.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:31.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:31.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:31.908 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:31.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:31.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:31.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:31.909 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:31.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:32.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:31.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:32.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:31.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:32.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:31.906+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:32.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:32.220+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46037 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:32.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:32.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:32.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:32.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:32.885 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:32.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:32.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:32.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:32.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:32.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:32.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:32.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:32.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:32.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:32.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:32.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:32.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:33.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:33.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:33.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:33.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:33.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:33.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:33.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:33.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:33.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:33.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:33.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:33.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:33.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:33.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:33.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:33.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:34.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:34.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:34.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:34.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:34.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:34.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:34.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:34.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:34.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:34.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:34.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:34.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:35.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:35.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:35.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:35.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:35.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:35.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:35.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:35.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:35.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:35.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:35.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:35.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:35.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:35.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:35.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:36.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:36.817+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:36.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:36.817+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:36.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:36.817+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:36.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:36.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:36.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:36.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:36.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:36.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:36.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:36.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:36.679+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:36.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:36.679+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:37.249 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:37.226+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46111 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:37.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:37.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:37.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:37.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:37.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:37.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:37.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:37.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:37.822+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:37.822+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:37.822+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:38.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:38.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:38.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:38.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:38.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:38.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:38.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:38.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:38.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:38.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:38.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:38.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:38.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:38.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:39.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:39.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:39.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:39.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:39.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:39.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:39.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:39.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:39.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:39.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:39.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:39.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:40.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:40.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:40.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:40.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:40.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:40.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:40.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:40.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:40.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:40.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:41.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:41.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:41.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:41.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:41.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:41.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:41.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:41.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:41.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:41.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:41.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:41.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:41.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:41.582+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:42.355 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:42.233+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46183 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:42.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:42.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:42.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:42.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:42.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:42.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:42.728+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:43.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:42.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:43.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:42.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:43.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:42.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:43.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:43.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:43.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:43.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:43.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:43.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:43.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:43.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:43.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:43.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:43.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:43.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:43.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:43.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:43.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:43.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:43.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:43.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:44.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:44.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:44.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:44.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:44.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:44.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:44.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:44.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:44.785+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:44.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:44.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:44.616+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:44.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:44.616+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:45.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:45.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:45.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:45.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:45.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:45.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:45.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:45.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:45.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:45.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:45.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:45.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:45.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:46.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:46.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:46.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:46.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:46.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:46.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:46.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:46.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:46.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:46.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:46.750+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:46.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:46.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:46.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:46.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:46.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:46.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:46.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:46.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:47.366 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:47.239+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46257 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:47.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:47.738+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:47.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:47.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:47.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:47.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:47.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:47.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:47.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:47.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:47.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:48.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:48.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:48.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:48.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:48.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:48.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:48.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:48.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:48.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:48.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:48.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:48.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:49.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:49.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:49.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:49.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:49.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:49.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:49.724+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:49.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:49.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:49.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:49.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:50.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:50.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:50.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:50.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:50.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:50.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:50.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:50.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:50.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:50.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:50.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:50.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:50.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:50.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:50.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:50.747+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:51.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:51.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:51.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:51.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:51.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:51.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:51.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:51.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:51.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:51.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:51.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:51.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:51.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:51.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:51.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:51.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:52.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:52.246+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46329 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:52.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:53.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:52.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:52.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:52.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:52.792+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:52.792+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:52.792+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:53.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:52.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:53.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:52.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:53.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:52.741+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:53.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:53.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:53.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:53.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:53.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:53.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:53.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:54.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:53.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:54.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:53.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:54.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:53.745+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:54.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:55.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:54.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:54.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:54.803+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:55.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:54.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:55.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:54.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:55.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:54.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:55.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:54.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:55.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:54.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:55.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:54.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:56.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:55.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:55.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:55.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:55.816+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:55.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:55.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:55.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:56.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:55.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:56.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:55.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:56.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:55.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:57.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:56.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:57.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:56.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:57.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:56.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:57.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:56.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:57.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:56.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:57.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:56.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:57.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:56.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:57.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:56.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:57.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:56.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:57.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:56.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:57.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:07:57.253+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46403 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:07:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:57.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:58.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:57.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:57.803+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:58.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:57.803+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:57.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:57.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:58.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:57.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:58.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:58.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:58.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:59.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:58.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:58.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:58.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:07:59.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:58.830+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:58.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:58.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:58.867+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:07:59.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:58.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:59.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:58.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:07:59.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:58.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:00.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:59.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:00.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:59.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:59.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:07:59.841+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:59.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:59.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:07:59.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:00.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:59.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:00.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:59.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:00.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:07:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:07:59.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:00.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:01.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:00.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:00.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:00.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:00.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:00.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:00.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:00.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:01.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:00.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:01.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:00.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:02.040 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:01.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:02.040 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:01.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:02.040 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:01.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:02.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:01.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:02.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:01.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:02.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:01.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:02.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:01.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:02.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:01.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:02.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:01.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:02.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:01.826+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:02.387 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:02.260+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46476 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:03.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:02.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:02.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:02.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:03.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:02.799+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:03.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:02.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:03.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:02.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:03.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:02.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:03.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:02.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:03.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:02.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:03.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:02.767+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:03.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:03.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:04.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:03.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:03.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:03.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:04.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:03.795+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:03.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:04.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:03.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:04.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:03.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:04.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:03.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:04.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:04.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:04.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:04.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:04.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:04.759+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:04.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:04.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:04.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:04.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:06.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:05.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:06.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:05.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:06.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:05.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:06.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:05.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:06.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:05.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:06.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:05.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:06.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:05.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:06.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:05.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:06.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:05.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:06.131 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:05.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:06.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:06.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:06.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:06.735+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:06.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:06.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:06.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:06.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:07.701 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:07.267+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46550 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:07.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:07.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:07.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:07.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:07.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:07.700+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:07.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:07.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:07.813+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:07.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:07.813+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:08.801+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:08.801+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:08.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:08.801+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:08.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:08.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:08.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:08.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:08.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:08.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:08.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:08.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:08.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:08.685+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:09.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:09.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:09.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:09.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:09.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:09.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:09.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:09.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:09.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:09.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:09.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:10.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:10.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:10.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:10.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:10.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:10.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:10.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:10.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:10.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:10.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:10.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:10.824+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:11.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:11.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:11.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:11.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:11.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:11.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:11.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:11.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:11.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:11.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:11.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:11.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:11.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:11.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:11.710+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:12.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:12.274+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46622 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:12.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:12.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:12.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:12.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:12.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:13.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:12.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:12.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:12.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:13.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:12.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:13.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:12.824+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:13.865+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:13.865+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:13.865+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:13.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:13.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:13.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:13.706+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:13.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:13.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:13.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:13.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:13.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:14.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:14.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:14.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:14.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:14.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:14.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:14.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:14.682+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:14.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:14.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:14.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:14.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:15.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:15.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:15.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:15.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:15.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:15.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:15.863+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:15.863+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:15.863+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:15.863+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:16.929 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:16.649+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:16.929 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:16.649+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:16.929 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:16.649+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:16.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:16.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:16.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:16.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:16.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:16.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:16.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:16.820+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:16.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:16.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:16.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:17.689 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:17.282+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46696 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:17.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:17.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:17.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:17.689+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:17.903+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:17.903+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:17.903+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:17.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:17.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:17.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:17.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:17.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:17.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:17.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:17.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:18.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:18.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:18.919 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:18.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:18.919 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:18.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:18.919 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:18.855+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:18.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:18.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:18.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:18.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:18.919 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:18.663+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:18.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:18.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:18.917+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:19.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:19.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:19.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:19.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:19.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:19.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:19.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:19.640+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:19.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:19.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:19.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:19.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:19.832+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:20.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:20.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:20.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:20.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:20.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:20.661+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:20.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:20.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:20.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:20.848+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:21.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:21.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:21.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:21.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:21.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:21.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:21.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:21.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:21.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:21.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:22.699 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:22.290+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46768 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:22.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:22.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:22.698+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:22.921+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:22.921+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:22.921+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:22.922+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:22.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:22.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:22.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:22.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:22.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:23.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:23.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:23.879+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:23.879+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:23.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:23.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:23.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:23.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:23.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:23.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:23.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:23.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:23.864+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:24.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:24.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:24.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:24.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:24.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:24.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:24.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:24.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:24.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:24.847+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:24.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:24.847+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:24.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:24.847+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:25.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:25.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:25.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:25.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:25.707+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:25.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:25.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:25.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:25.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:25.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:25.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:25.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:25.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:25.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:25.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:26.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:26.662+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:26.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:26.662+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:26.924 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:26.662+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:26.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:26.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:26.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:26.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:26.925 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:26.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:26.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:26.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:26.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:26.922+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:27.687 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:27.296+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46842 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:27.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:27.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:27.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:27.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:27.909+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:27.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:27.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:27.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:27.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:27.891+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:27.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:27.891+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:27.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:27.891+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:28.939 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:28.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:28.939 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:28.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:28.939 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:28.676+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:29.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:28.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:28.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:28.956+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:28.956+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:28.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:28.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:29.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:28.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:29.655+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:29.655+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:29.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:29.655+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:29.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:29.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:29.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:30.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:29.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:30.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:29.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:30.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:29.975+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:30.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:29.975+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:30.930 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:30.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:30.930 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:30.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:30.930 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:30.606+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:31.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:30.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:30.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:30.928+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:31.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:30.928+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:30.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:30.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:30.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:31.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:31.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:31.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:31.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:31.890 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:31.633+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:32.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:31.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:32.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:31.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:32.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:31.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:32.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:31.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:31.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:31.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:31.932+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:32.671 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:32.302+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46914 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:32.933 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:32.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:32.933 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:32.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:32.933 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:32.670+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:32.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:33.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:32.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:32.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:32.931+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:32.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:32.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:32.937+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:33.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:33.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:33.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:33.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:33.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:33.645+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:33.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:33.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:33.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:33.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:33.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:34.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:33.940+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:33.940+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:34.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:33.940+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:34.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:34.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:34.905+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:34.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:34.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:34.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:34.852+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:34.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:34.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:34.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:34.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:34.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:34.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:35.915 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:35.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:35.916 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:35.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:35.916 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:35.657+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:35.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:35.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:35.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:35.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:35.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:35.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:35.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:35.809+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:35.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:35.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:36.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:35.913+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:36.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:36.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:36.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:36.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:36.699+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:36.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:36.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:36.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:36.776+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:36.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:36.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:36.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:36.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:36.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:36.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:37.722 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:37.310+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 46988 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:37.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:37.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:37.719+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:37.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:37.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:37.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:37.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:37.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:37.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:37.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:37.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:37.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:37.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:37.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:37.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:38.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:38.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:38.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:38.704+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:38.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:38.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:38.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:38.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:38.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:38.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:38.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:38.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:38.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:38.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:39.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:39.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:39.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:39.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:39.737+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:39.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:39.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:39.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:39.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:39.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:39.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:39.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:39.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:39.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:39.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:40.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:40.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:40.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:40.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:40.754+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:40.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:40.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:40.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:40.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:40.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:40.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:40.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:40.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:40.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:40.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:40.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:41.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:41.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:41.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:41.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:41.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:41.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:41.988 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:41.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:41.988 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:41.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:41.989 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:41.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:41.989 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:41.786+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:42.320 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:41.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:42.320 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:41.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:42.321 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:41.986+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:42.692 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:42.318+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47061 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:42.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:42.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:42.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:42.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:42.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:42.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:42.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:42.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:42.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:42.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:42.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:42.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:42.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:42.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:42.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:42.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:43.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:42.942+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:43.931+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:43.931+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:43.931+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:43.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:43.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:43.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:43.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:43.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:43.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:43.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:44.964 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:44.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:44.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:44.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:44.965 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:44.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:44.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:44.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:44.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:44.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:44.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:44.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:44.965 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:44.683+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:44.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:44.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:44.963+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:45.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:45.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:45.930+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:45.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:45.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:45.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:45.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:45.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:45.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:45.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:45.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:45.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:45.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:47.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:46.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:47.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:46.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:46.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:47.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:46.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:47.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:46.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:47.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:46.853+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:46.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:46.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:46.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:47.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:46.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:47.588 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:47.325+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47136 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:47.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:47.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:47.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:47.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:47.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:47.750+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:47.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:47.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:47.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:48.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:47.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:47.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:47.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:47.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:47.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:48.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:48.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:48.922+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:48.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:48.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:49.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:48.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:48.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:48.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:49.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:48.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:49.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:48.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:49.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:49.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:49.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:49.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:49.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:50.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:49.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:49.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:50.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:49.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:50.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:49.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:50.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:49.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:50.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:50.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:50.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:50.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:50.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:50.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:50.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:50.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:51.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:50.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:51.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:50.843+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:51.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:51.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:52.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:51.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:51.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:51.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:52.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:51.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:52.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:52.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:51.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:52.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:51.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:52.630 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:52.333+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47209 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:52.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:52.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:52.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:52.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:52.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:52.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:52.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:52.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:53.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:52.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:52.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:53.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:53.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:54.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:53.996+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:53.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:53.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:54.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:53.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:54.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:53.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:54.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:53.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:54.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:53.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:54.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:53.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:54.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:54.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:55.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:54.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:54.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:54.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:55.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:54.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:54.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:54.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:54.843+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:55.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:54.843+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:55.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:55.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:55.993+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:55.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:55.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:55.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:55.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:55.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:55.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:56.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:55.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:56.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:56.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:56.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:56.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:56.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:56.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:56.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:56.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:56.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:57.343 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:57.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:57.343 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:57.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:57.343 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:57.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:57.672 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:08:57.341+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47282 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:08:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:57.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:57.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:57.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:57.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:57.813+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:57.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:57.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:57.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:57.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:58.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:58.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:58.003+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:58.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:58.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:59.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:58.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:08:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:59.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:59.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:59.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:08:59.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:08:59.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:58.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:59.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:58.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:59.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:58.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:08:59.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:58.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:59.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:59.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:00.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:08:59.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:00.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:00.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:00.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:00.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:59.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:00.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:59.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:00.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:59.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:00.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:08:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:08:59.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:01.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:01.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:01.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:01.061+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:00.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:00.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:00.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:00.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:01.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:00.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:01.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:00.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:01.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:00.823+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:02.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:01.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:02.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:01.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:02.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:01.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:02.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:01.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:02.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:01.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:02.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:01.864+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:02.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:01.864+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:02.350 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:02.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:02.350 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:02.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:02.350 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:02.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:02.714 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:02.348+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47355 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:02.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:02.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:02.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:02.798+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:02.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:02.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:02.824+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:02.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:02.825+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:03.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:03.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:03.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:04.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:03.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:04.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:03.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:04.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:03.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:04.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:03.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:04.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:03.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:04.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:03.817+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:04.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:03.817+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:04.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:04.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:04.111+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:05.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:05.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:05.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:04.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:04.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:05.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:04.883+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:05.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:04.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:04.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:04.825+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:05.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:04.825+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:06.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:06.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:06.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:06.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:06.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:06.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:06.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:05.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:06.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:05.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:06.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:05.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:05.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:05.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:05.816+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:05.816+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:07.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:07.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:07.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:06.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:06.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:07.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:06.849+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:07.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:06.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:06.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:06.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:07.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:06.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:07.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:07.356+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47428 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:07.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:07.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:07.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:07.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:07.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:08.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:07.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:07.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:07.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:07.790+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:07.790+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:09.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:08.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:08.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:08.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:09.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:08.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:09.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:08.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:09.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:08.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:09.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:08.979+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:09.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:08.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:09.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:08.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:09.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:08.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:09.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:09.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:09.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:09.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:10.020+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:10.020+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:10.020+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:10.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:09.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:10.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:09.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:10.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:09.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:11.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:10.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:11.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:10.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:11.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:10.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:11.040 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:10.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:11.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:10.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:11.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:10.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:11.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:10.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:11.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:11.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:11.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:12.046 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:11.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:12.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:11.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:12.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:11.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:12.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:11.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:12.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:11.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:12.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:11.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:12.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:11.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:12.364 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:12.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:12.364 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:12.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:12.365 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:12.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:12.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:12.363+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47501 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:13.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:12.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:13.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:12.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:13.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:12.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:13.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:12.771+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:13.045 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:12.902+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:13.045 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:12.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:13.046 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:12.903+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:13.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:13.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:13.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:14.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:13.931+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:14.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:13.931+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:14.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:13.931+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:14.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:13.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:14.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:13.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:14.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:13.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:14.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:13.758+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:14.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:14.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:14.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:15.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:14.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:15.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:14.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:15.061 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:14.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:15.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:14.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:15.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:14.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:15.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:14.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:15.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:14.801+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:15.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:15.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:15.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:16.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:16.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:16.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:16.044+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:15.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:15.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:16.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:15.991+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:15.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:16.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:15.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:16.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:15.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:16.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:15.839+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:17.003+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:17.003+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:17.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:17.003+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:17.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:17.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:17.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:17.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:16.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:17.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:16.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:17.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:16.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:17.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:16.799+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:17.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:17.369+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47574 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:18.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:18.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:18.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:18.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:18.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:18.021+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:18.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:17.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:18.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:17.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:18.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:17.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:18.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:17.815+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:18.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:18.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:19.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:18.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:19.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:19.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:19.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:19.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:19.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:18.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:18.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:18.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:19.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:18.852+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:20.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:19.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:20.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:19.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:20.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:19.927+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:20.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:19.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:20.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:19.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:20.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:19.857+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:20.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:19.857+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:20.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:20.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:20.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:20.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:20.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:21.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:20.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:21.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:21.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:21.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:21.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:21.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:21.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:21.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:20.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:21.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:20.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:21.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:20.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:21.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:20.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:22.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:21.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:22.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:21.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:22.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:21.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:22.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:21.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:22.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:21.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:22.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:21.853+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:22.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:21.853+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:22.378 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:22.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:22.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:22.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:22.379 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:22.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:22.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:22.376+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47647 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:23.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:22.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:23.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:22.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:23.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:22.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:23.168 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:22.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:23.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:22.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:23.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:22.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:23.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:22.962+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:23.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:23.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:23.165+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:24.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:23.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:24.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:23.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:24.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:23.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:24.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:23.905+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:24.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:23.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:24.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:23.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:24.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:23.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:24.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:24.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:24.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:24.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:25.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:24.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:25.208 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:24.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:25.208 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:24.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:25.208 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:24.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:25.208 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:24.985+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:25.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:24.985+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:25.209 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:24.985+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:25.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:25.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:25.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:26.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:25.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:26.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:25.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:26.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:25.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:26.216 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:25.871+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:26.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:25.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:26.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:25.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:26.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:25.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:26.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:26.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:26.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:26.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:26.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:27.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:26.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:26.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:27.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:26.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:27.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:26.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:27.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:26.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:27.509 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:27.384+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47720 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:27.510 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:27.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:27.510 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:27.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:27.510 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:27.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:28.206 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:27.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:28.206 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:27.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:28.207 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:27.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:28.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:27.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:28.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:27.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:28.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:27.779+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:28.207 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:27.779+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:28.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:28.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:28.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:28.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:28.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:28.969+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:28.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:28.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:28.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:28.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:29.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:29.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:29.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:29.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:29.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:29.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:29.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:29.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:30.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:29.989+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:30.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:29.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:30.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:29.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:30.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:29.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:30.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:29.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:30.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:30.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:30.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:30.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:30.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:30.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:30.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:30.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:30.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:30.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:30.984 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:30.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:30.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:30.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:30.982+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:31.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:31.197+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:31.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:31.197+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:31.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:31.197+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:31.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:31.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:31.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:31.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:31.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:32.393 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:32.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:32.394 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:32.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:32.394 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:32.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:32.394 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:32.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:32.394 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:32.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:32.394 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:32.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:32.723 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:32.391+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47793 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:32.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:32.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:32.721+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:32.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:33.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:33.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:33.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:33.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:33.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:33.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:33.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:34.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:33.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:34.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:33.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:34.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:33.770+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:34.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:33.770+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:34.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:34.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:34.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:34.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:34.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:34.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:34.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:35.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:34.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:35.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:34.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:35.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:34.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:35.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:34.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:35.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:35.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:35.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:35.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:35.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:35.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:36.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:35.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:36.046 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:35.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:36.046 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:35.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:36.046 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:35.761+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:36.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:36.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:36.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:36.043+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:36.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:36.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:36.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:36.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:36.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:36.080+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:36.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:36.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:36.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:36.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:36.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:37.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:37.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:37.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:37.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:37.401 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:37.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:37.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:37.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:37.402 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:37.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:37.402 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:37.099+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:37.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:37.398+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47866 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:38.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:37.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:38.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:37.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:38.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:37.761+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:38.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:37.761+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:38.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:38.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:38.102+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:38.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:38.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:38.045+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:39.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:38.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:39.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:38.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:39.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:38.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:39.052 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:38.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:39.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:39.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:39.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:39.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:39.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:39.062+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:40.036 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:39.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:40.036 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:39.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:40.036 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:39.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:40.036 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:39.754+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:40.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:40.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:40.058+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:40.034+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:40.034+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:40.034+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:41.038 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:40.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:41.038 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:40.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:41.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:40.773+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:41.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:40.773+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:41.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:41.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:41.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:41.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:41.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:41.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:42.047+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:42.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:42.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:42.030+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:41.820+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:41.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:42.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:41.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:42.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:41.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:42.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:42.407+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 47939 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:43.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:42.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:43.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:42.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:43.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:42.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:43.042 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:42.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:43.042 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:42.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:43.043 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:42.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:43.043 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:42.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:43.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:43.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:43.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:44.074 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:43.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:44.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:43.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:44.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:43.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:44.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:43.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:44.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:43.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:44.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:43.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:44.075 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:43.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:44.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:44.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:44.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:45.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:44.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:45.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:44.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:45.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:44.949+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:44.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:44.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:44.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:45.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:44.772+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:45.109+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:45.109+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:45.109+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:46.143 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:45.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:46.143 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:45.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:46.144 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:45.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:46.144 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:45.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:46.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:45.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:46.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:45.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:46.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:45.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:46.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:46.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:46.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:46.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:46.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:46.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:46.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:47.415 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:46.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:47.415 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:46.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:47.415 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:46.980+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:47.416 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:47.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:47.416 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:47.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:47.416 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:47.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:47.717 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:47.414+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48012 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:47.946+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:47.946+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:47.946+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:47.716+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:47.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:48.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:48.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:48.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:48.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:48.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:48.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:48.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:48.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:48.985+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:48.985+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:48.985+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:49.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:49.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:49.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:49.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:49.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:49.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:49.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:50.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:50.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:50.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:50.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:50.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:50.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:50.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:50.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:50.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:50.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:51.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:51.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:51.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:51.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:51.026+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:51.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:51.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:51.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:51.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:51.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:51.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:51.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:51.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:51.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:51.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:52.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:52.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:52.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:52.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:52.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:52.075+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:52.423 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:52.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:52.423 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:52.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:52.423 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:52.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:52.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:52.421+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48085 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:52.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:52.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:52.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:52.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:52.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:53.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:53.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:53.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:53.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:53.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:53.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:53.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:53.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:53.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:53.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:53.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:54.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:54.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:54.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:54.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:54.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:54.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:54.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:54.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:54.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:54.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:55.246 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:55.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:55.247 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:55.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:55.247 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:55.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:55.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:55.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:55.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:55.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:55.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:55.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:55.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:55.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:55.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:55.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:56.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:56.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:56.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:56.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:56.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:56.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:56.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:56.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:56.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:56.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:56.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:56.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:56.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:57.430 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:57.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:57.430 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:57.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:57.430 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:57.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:57.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:57.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:57.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:57.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:57.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:57.286+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:57.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:09:57.429+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48158 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:09:57.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:57.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:57.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:57.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:57.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:58.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:58.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:58.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:58.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:58.255+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:58.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:58.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:58.095+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:58.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:58.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:58.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:58.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:59.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:59.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:09:59.120+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:09:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:59.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:59.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:09:59.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:09:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:59.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:59.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:59.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:59.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:09:59.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:09:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:09:59.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:00.146+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:00.146+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:00.146+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:00.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:00.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:00.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:00.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:00.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:00.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:00.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:00.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:00.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:00.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:00.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:01.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:01.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:01.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:01.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:01.291+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:01.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:01.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:01.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:01.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:01.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:01.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:01.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:02.438 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:02.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:02.438 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:02.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:02.438 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:02.142+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:02.438 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:02.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:02.438 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:02.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:02.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:02.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:02.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:02.436+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48231 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:02.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:02.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:02.603+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:02.603+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:03.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:03.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:03.121+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:03.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:03.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:03.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:03.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:03.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:03.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:03.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:03.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:04.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:04.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:04.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:04.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:04.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:04.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:04.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:04.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:04.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:04.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:04.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:04.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:05.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:05.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:05.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:05.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:05.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:05.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:05.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:05.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:05.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:05.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:05.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:06.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:06.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:06.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:06.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:06.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:06.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:06.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:06.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:06.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:06.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:06.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:06.672+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:06.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:07.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:07.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:07.446 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:07.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:07.446 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:07.063+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:07.446 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:07.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:07.446 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:07.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:07.446 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:07.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:07.700 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:07.443+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48304 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:07.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:07.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:07.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:07.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:07.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:07.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:08.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:08.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:08.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:08.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:08.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:08.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:08.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:08.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:08.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:08.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:09.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:08.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:09.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:08.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:09.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:08.731+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:09.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:08.731+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:09.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:09.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:09.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:09.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:09.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:09.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:09.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:09.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:10.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:09.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:10.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:09.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:10.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:09.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:10.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:09.754+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:10.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:10.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:10.137+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:10.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:10.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:10.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:11.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:10.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:11.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:10.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:11.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:10.801+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:11.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:10.801+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:11.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:11.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:11.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:11.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:11.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:11.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:12.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:11.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:12.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:11.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:12.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:11.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:12.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:11.808+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:12.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:12.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:12.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:12.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:12.454 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:12.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:12.454 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:12.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:12.454 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:12.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:12.454 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:12.090+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:12.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:12.451+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48377 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:13.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:12.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:13.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:12.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:13.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:12.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:13.104 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:12.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:13.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:13.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:13.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:13.160+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:13.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:13.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:13.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:14.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:13.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:14.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:13.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:14.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:13.781+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:14.083 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:13.781+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:14.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:14.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:14.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:14.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:14.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:14.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:14.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:14.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:14.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:14.081+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:15.034 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:14.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:15.034 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:14.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:15.034 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:14.737+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:15.035 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:14.737+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:15.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:15.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:15.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:15.126+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:15.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:15.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:15.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:15.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:15.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:15.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:15.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:15.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:16.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:16.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:16.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:16.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:16.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:16.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:16.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:16.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:16.061+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:16.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:16.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:16.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:16.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:16.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:16.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:17.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:17.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:17.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:17.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:17.461 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:17.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:17.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:17.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:17.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:17.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:17.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:17.051+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:17.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:17.458+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48450 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:18.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:17.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:18.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:17.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:18.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:17.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:18.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:17.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:18.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:18.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:18.092+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:18.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:18.095+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:18.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:18.095+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:18.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:18.095+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:19.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:18.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:19.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:18.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:19.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:18.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:19.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:18.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:19.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:19.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:19.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:19.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:19.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:19.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:20.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:19.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:20.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:19.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:20.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:19.741+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:20.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:19.741+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:20.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:20.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:20.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:20.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:20.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:20.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:20.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:20.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:20.082+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:21.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:20.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:21.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:20.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:21.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:20.782+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:21.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:20.782+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:21.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:21.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:21.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:21.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:21.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:21.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:21.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:21.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:21.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:22.042 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:21.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:22.042 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:21.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:22.042 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:21.744+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:22.042 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:21.744+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:22.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:22.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:22.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:22.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:22.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:22.054+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:22.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:22.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:22.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:22.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:22.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:22.039+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:22.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:22.465+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48523 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:23.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:23.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:23.005+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:22.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:22.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:22.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:23.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:22.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:23.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:23.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:23.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:23.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:23.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:23.002+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:24.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:23.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:24.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:23.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:24.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:23.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:24.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:23.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:24.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:24.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:24.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:24.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:24.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:24.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:24.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:24.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:24.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:25.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:25.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:25.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:25.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:25.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:25.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:25.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:25.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:25.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:25.031+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:24.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:24.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:25.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:24.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:25.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:24.787+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:26.044 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:25.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:26.044 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:25.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:26.044 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:25.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:26.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:25.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:26.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:25.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:26.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:25.771+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:26.045 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:25.771+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:26.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:26.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:26.042+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:27.081 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:26.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:27.081 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:26.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:27.081 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:26.998+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:27.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:26.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:27.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:26.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:27.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:26.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:27.082 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:26.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:27.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:27.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:27.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:27.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:27.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:27.079+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:27.730 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:27.472+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48596 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:28.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:27.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:28.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:27.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:28.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:27.775+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:28.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:27.775+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:28.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:27.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:28.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:27.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:28.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:27.951+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:28.424 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:28.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:28.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:28.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:28.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:28.049+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:29.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:28.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:29.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:28.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:29.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:28.938+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:29.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:28.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:29.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:28.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:29.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:28.747+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:29.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:28.747+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:29.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:29.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:29.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:30.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:29.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:30.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:29.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:30.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:29.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:30.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:30.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:30.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:30.009+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:30.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:29.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:30.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:29.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:30.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:29.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:30.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:29.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:30.884+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:31.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:30.884+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:31.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:30.884+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:31.017+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:31.017+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:31.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:31.017+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:30.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:30.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:30.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:31.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:30.804+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:32.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:31.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:31.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:32.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:31.836+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:32.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:32.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:32.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:32.064+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:31.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:31.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:31.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:32.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:31.837+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:32.842 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:32.481+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48669 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:33.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:32.840+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:32.840+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:33.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:32.840+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:32.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:32.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:32.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:33.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:32.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:33.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:33.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:33.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:33.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:33.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:33.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:34.073 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:33.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:34.073 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:33.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:34.073 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:33.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:34.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:33.809+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:34.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:33.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:34.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:33.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:34.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:33.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:34.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:34.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:34.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:35.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:34.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:35.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:34.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:35.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:34.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:35.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:34.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:35.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:34.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:35.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:34.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:35.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:34.793+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:35.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:35.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:35.058+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:36.084 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:35.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:36.084 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:35.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:36.084 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:35.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:36.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:35.777+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:36.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:35.777+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:36.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:35.777+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:36.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:35.777+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:36.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:36.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:36.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:37.061 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:36.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:37.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:36.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:37.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:36.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:37.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:36.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:37.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:36.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:37.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:36.799+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:37.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:36.800+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:37.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:37.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:37.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:37.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:37.841 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:37.488+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48742 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:38.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:37.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:38.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:37.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:38.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:37.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:38.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:38.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:38.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:38.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:37.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:37.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:37.839+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:38.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:37.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:39.088 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:38.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:39.088 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:38.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:39.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:38.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:39.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:38.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:39.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:38.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:39.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:38.808+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:39.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:38.808+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:39.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:39.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:39.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:39.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:39.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:39.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:40.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:40.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:40.083+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:40.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:39.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:40.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:39.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:40.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:39.855+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:40.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:39.855+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:40.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:41.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:40.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:41.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:40.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:40.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:40.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:41.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:40.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:41.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:40.904+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:41.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:41.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:41.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:41.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:41.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:41.108+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:41.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:42.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:41.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:42.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:41.925+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:41.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:41.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:41.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:42.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:41.938+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:42.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:42.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:42.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:42.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:42.230 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:42.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:42.920 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:42.496+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48815 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:43.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:42.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:43.193 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:42.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:43.193 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:42.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:43.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:42.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:43.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:42.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:43.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:42.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:43.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:42.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:43.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:43.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:43.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:44.225 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:43.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:44.225 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:43.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:44.225 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:43.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:44.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:43.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:44.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:43.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:44.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:43.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:44.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:43.949+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:44.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:44.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:44.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:44.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:44.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:44.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:45.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:44.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:45.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:44.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:45.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:44.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:45.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:44.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:45.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:45.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:45.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:45.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:45.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:45.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:45.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:45.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:46.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:45.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:45.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:45.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:45.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:45.909+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:46.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:46.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:46.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:47.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:46.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:47.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:46.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:47.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:46.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:46.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:46.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:46.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:46.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:47.630 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:47.504+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48887 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:47.630 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:47.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:47.630 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:47.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:47.630 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:47.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:47.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:47.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:47.865+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:47.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:47.865+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:48.302 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:47.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:48.302 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:47.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:48.302 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:47.990+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:48.593 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:48.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:48.594 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:48.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:48.594 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:48.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:49.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:48.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:48.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:49.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:48.950+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:48.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:48.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:48.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:48.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:49.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:49.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:49.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:49.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:49.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:50.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:49.988+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:49.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:49.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:49.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:50.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:49.916+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:50.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:50.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:50.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:50.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:50.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:50.968+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:50.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:50.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:50.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:51.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:50.949+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:51.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:51.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:51.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:51.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:52.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:51.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:52.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:51.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:51.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:51.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:52.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:51.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:52.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:51.900+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:52.634 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:52.511+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 48961 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:52.634 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:52.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:52.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:52.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:52.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:52.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:52.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:52.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:52.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:52.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:52.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:52.852+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:52.963 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:52.852+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:52.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:52.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:53.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:52.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:53.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:53.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:53.636 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:53.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:53.636 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:53.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:53.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:53.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:53.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:53.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:53.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:53.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:53.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:53.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:53.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:54.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:53.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:54.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:53.936+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:54.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:54.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:54.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:54.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:54.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:54.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:54.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:54.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:54.976+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:54.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:54.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:54.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:55.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:54.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:55.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:55.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:55.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:56.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:56.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:56.013+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:55.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:55.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:55.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:56.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:55.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:56.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:56.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:56.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:56.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:56.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:56.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:56.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:56.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:56.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:56.854+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:56.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:56.854+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:57.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:57.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:57.338 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:57.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:57.338 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:57.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:57.644 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:10:57.519+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49033 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:10:57.644 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:57.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:57.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:57.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:57.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:57.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:57.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:57.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:57.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:57.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:57.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:57.858+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:58.375 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:58.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:58.375 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:58.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:58.375 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:58.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:58.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:58.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:58.678 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:58.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:58.678 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:58.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:58.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:58.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:58.858+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:10:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:58.859+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:59.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:59.084+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:59.348 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:59.084+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:59.349 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:10:59.084+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:10:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:59.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:59.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:10:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:10:59.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:00.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:00.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:00.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:59.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:59.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:59.881+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:00.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:10:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:10:59.881+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:00.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:00.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:00.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:01.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:01.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:01.115+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:00.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:00.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:00.886+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:01.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:00.886+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:01.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:01.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:01.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:02.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:02.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:02.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:02.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:01.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:01.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:01.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:02.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:01.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:02.549 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:02.525+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49107 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:02.550 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:02.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:02.550 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:02.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:02.550 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:02.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:03.157 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:02.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:03.158 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:02.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:03.158 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:02.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:03.158 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:02.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:03.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:03.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:03.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:03.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:03.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:03.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:03.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:03.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:03.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:03.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:03.889+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:04.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:04.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:04.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:04.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:04.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:04.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:05.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:04.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:05.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:04.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:05.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:04.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:05.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:04.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:05.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:05.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:05.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:05.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:05.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:05.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:05.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:05.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:06.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:05.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:06.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:05.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:06.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:05.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:06.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:06.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:06.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:06.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:06.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:06.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:06.289+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:07.084 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:06.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:07.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:06.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:07.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:06.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:07.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:06.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:07.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:07.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:07.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:07.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:07.082+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:07.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:07.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:07.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:07.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:07.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:07.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:07.830 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:07.533+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49179 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:08.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:08.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:08.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:08.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:08.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:08.054+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:07.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:08.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:07.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:07.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:08.230 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:07.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:08.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:08.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:08.259+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:09.016 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:08.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:09.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:08.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:09.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:08.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:09.017 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:08.849+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:09.295 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:09.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:09.295 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:09.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:09.295 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:09.015+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:09.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:09.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:09.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:09.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:09.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:10.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:09.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:10.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:09.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:10.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:10.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:10.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:10.008+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:10.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:10.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:10.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:10.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:10.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:10.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:11.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:10.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:10.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:10.991+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:10.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:10.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:10.806+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:10.806+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:11.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:11.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:11.265+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:12.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:12.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:12.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:12.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:11.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:11.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:11.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:12.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:11.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:12.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:12.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:12.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:12.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:12.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:12.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:12.799 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:12.541+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49253 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:13.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:13.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:13.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:13.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:13.005+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:12.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:12.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:12.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:13.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:12.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:13.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:13.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:13.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:13.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:13.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:13.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:14.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:13.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:14.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:13.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:14.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:13.978+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:13.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:13.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:13.772+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:14.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:13.772+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:14.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:14.338+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:14.339+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:15.028 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:14.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:15.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:14.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:15.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:14.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:15.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:14.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:15.368 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:15.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:15.368 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:15.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:15.368 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:15.027+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:15.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:15.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:15.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:15.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:15.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:15.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:15.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:15.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:15.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:15.985 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:15.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:16.364 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:15.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:16.365 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:15.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:16.365 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:15.983+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:16.704 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:16.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:16.704 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:16.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:16.704 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:16.362+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:16.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:16.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:16.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:16.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:16.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:16.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:16.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:16.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:16.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:16.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:16.974+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:17.676 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:17.548+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49325 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:17.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:17.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:17.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:17.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:17.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:17.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:17.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:17.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:17.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:17.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:17.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:17.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:17.962 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:17.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:17.960+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:18.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:17.960+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:18.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:17.960+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:18.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:18.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:18.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:18.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:18.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:18.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:18.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:18.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:18.730+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:18.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:18.730+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:19.389 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:18.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:19.389 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:18.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:19.389 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:18.981+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:19.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:19.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:19.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:19.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:19.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:19.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:19.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:19.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:19.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:19.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:19.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:19.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:19.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:19.683+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:19.946+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:19.946+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:19.946+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:20.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:20.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:20.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:20.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:20.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:20.658+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:20.658+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:20.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:20.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:20.953+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:21.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:21.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:21.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:21.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:21.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:21.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:21.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:21.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:22.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:21.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:22.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:21.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:22.680 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:22.557+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49399 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:22.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:22.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:22.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:22.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:22.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:22.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:22.919+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:22.919+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:22.919+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:22.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:22.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:22.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:22.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:23.697 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:23.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:23.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:23.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:23.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:23.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:23.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:23.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:23.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:23.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:23.921+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:23.921+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:23.921+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:24.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:24.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:24.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:24.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:24.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:24.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:24.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:24.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:24.880+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:24.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:24.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:24.692+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:24.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:25.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:25.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:25.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:25.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:25.722 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:25.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:25.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:25.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:25.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:25.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:25.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:25.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:25.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:25.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:26.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:26.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:26.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:27.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:26.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:27.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:26.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:27.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:26.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:26.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:26.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:26.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:27.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:26.730+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:27.687 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:27.564+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49471 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:27.688 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:27.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:27.688 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:27.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:27.688 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:27.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:27.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:27.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:27.879+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:27.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:27.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:27.739+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:27.739+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:28.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:28.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:28.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:28.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:29.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:28.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:29.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:28.838+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:28.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:28.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:28.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:29.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:28.757+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:29.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:29.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:29.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:29.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:29.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:29.423+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:29.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:29.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:29.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:29.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:29.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:29.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:29.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:30.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:30.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:30.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:30.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:30.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:30.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:30.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:30.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:30.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:30.869+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:30.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:30.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:30.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:30.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:31.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:31.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:31.656 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:31.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:31.657 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:31.375+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:31.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:31.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:31.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:31.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:31.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:31.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:31.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:31.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:32.630 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:32.571+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49545 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:32.630 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:32.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:32.630 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:32.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:32.631 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:32.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:32.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:32.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:32.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:32.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:32.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:32.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:32.801+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:32.801+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:32.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:32.801+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:33.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:33.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:33.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:33.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:33.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:33.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:33.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:33.833+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:33.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:33.833+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:33.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:33.833+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:33.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:33.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:33.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:33.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:33.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:33.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:33.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:33.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:34.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:34.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:34.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:34.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:34.640 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:34.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:34.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:34.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:34.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:34.798+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:34.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:34.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:34.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:34.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:35.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:35.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:35.646 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:35.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:35.647 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:35.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:35.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:35.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:35.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:35.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:35.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:35.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:35.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:35.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:35.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:36.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:36.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:36.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:36.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:36.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:36.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:36.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:36.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:36.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:36.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:36.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:36.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:36.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:36.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:36.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:36.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:36.612+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:37.697 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:37.579+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49617 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:37.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:37.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:37.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:37.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:37.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:37.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:37.698 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:37.590+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:37.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:37.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:37.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:37.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:37.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:37.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:37.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:37.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:37.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:37.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:38.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:38.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:38.376+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:38.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:38.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:38.604+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:38.604+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:39.013 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:38.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:39.014 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:38.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:39.014 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:38.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:39.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:39.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:39.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:39.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:39.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:39.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:39.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:39.622+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:39.622+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:40.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:39.767+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:39.767+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:40.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:39.767+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:40.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:40.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:40.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:40.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:40.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:40.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:40.593+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:40.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:40.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:41.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:40.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:41.596 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:41.324+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:41.596 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:41.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:41.596 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:41.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:41.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:41.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:41.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:41.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:41.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:41.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:41.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:42.587 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:42.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:42.587 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:42.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:42.588 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:42.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:42.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:42.586+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49691 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:42.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:42.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:42.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:42.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:42.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:42.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:42.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:42.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:43.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:43.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:43.629 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:43.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:43.629 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:43.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:43.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:43.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:43.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:43.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:43.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:43.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:43.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:43.627+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:44.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:44.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:44.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:44.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:44.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:44.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:44.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:44.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:44.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:44.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:44.602+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:44.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:45.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:44.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:45.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:44.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:45.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:45.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:45.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:45.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:45.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:45.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:45.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:45.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:45.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:45.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:45.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:45.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:45.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:45.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:46.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:46.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:46.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:46.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:46.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:46.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:46.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:46.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:46.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:46.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:46.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:46.602+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:46.602+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:47.614 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:47.592+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49763 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:47.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:47.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:47.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:47.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:47.615 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:47.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:47.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:47.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:47.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:47.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:47.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:47.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:47.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:47.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:47.627+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:48.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:48.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:48.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:48.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:48.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:48.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:48.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:49.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:48.743+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:49.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:48.743+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:49.098 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:48.743+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:49.351+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:49.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:49.351+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:49.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:49.351+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:49.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:49.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:49.584+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:49.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:49.585+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:49.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:49.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:50.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:49.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:50.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:50.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:50.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:50.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:50.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:50.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:50.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:50.593+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:50.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:51.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:50.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:51.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:50.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:51.385+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:51.385+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:51.385+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:51.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:51.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:51.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:51.586+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:51.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:51.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:52.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:51.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:52.723 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:52.600+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49837 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:52.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:52.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:52.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:52.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:52.724 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:52.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:52.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:52.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:52.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:52.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:52.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:52.554+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:52.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:52.554+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:52.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:52.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:52.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:53.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:53.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:53.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:53.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:53.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:53.551+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:53.551+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:54.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:53.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:54.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:53.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:54.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:53.856+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:54.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:54.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:54.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:54.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:54.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:54.517+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:54.517+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:54.517+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:54.517+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:54.889+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:54.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:55.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:54.890+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:55.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:55.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:55.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:55.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:55.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:55.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:55.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:55.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:55.550+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:55.858+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:56.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:55.858+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:56.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:55.858+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:56.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:56.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:56.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:56.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:56.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:56.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:56.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:56.505+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:56.505+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:56.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:56.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:57.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:56.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:57.627 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:11:57.606+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49909 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:11:57.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:57.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:57.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:57.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:57.628 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:57.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:57.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:57.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:57.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:57.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:57.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:57.480+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:57.629 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:57.480+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:57.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:57.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:57.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:57.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:57.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:58.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:58.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:58.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:58.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:58.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:58.510+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:58.510+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:59.181 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:58.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:59.181 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:58.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:59.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:58.910+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:11:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:59.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:59.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:11:59.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:11:59.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:59.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:59.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:59.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:59.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:59.530+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:11:59.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:11:59.530+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:59.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:59.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:00.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:11:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:11:59.948+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:00.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:00.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:00.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:00.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:00.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:00.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:00.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:00.508+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:00.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:00.508+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:00.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:00.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:01.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:00.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:01.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:01.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:01.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:01.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:01.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:01.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:01.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:01.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:01.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:02.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:01.947+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:02.617 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:02.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:02.617 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:02.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:02.617 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:02.349+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:02.617 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:02.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:02.617 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:02.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:02.618 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:02.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:02.618 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:02.520+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:02.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:02.614+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 49983 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:02.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:02.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:03.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:02.958+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:03.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:03.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:03.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:03.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:03.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:03.485+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:03.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:03.485+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:03.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:03.485+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:03.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:03.485+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:04.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:03.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:04.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:03.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:04.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:03.945+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:04.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:04.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:04.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:04.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:04.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:04.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:04.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:04.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:04.475+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:04.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:05.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:04.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:05.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:04.909+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:05.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:05.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:05.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:05.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:05.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:05.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:05.433+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:05.433+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:06.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:05.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:06.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:05.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:06.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:05.907+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:06.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:06.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:06.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:06.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:06.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:06.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:06.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:06.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:06.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:07.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:06.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:07.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:06.901+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:07.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:07.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:07.624 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:07.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:07.625 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:07.256+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:07.625 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:07.461+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:07.625 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:07.461+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:07.625 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:07.461+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:07.626 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:07.461+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:07.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:07.621+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50055 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:07.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:07.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:07.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:07.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:07.852+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:08.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:08.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:08.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:08.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:08.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:08.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:08.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:08.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:08.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:08.235+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:09.220 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:08.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:09.220 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:08.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:09.220 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:08.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:09.217+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:09.217+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:09.217+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:09.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:09.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:09.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:09.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:09.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:09.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:09.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:09.477+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:10.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:09.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:10.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:09.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:10.213 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:09.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:10.211+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:10.211+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:10.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:10.211+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:10.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:10.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:10.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:10.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:10.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:10.496+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:10.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:10.496+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:11.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:10.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:11.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:10.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:11.206 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:10.887+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:11.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:11.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:11.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:11.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:11.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:11.486+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:11.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:11.486+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:11.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:11.486+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:11.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:11.486+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:12.211 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:11.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:12.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:11.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:12.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:11.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:12.461 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:12.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:12.462 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:12.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:12.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:12.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:12.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:12.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:12.462 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:12.210+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:12.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:12.630+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50129 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:12.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:12.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:13.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:12.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:13.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:12.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:13.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:12.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:13.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:13.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:13.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:13.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:13.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:13.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:13.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:13.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:13.441+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:13.441+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:14.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:13.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:14.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:13.893+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:14.212 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:13.894+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:14.211+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:14.211+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:14.211+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:14.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:14.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:14.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:14.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:14.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:14.490+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:14.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:14.490+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:14.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:14.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:15.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:14.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:15.525 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:15.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:15.525 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:15.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:15.525 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:15.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:15.861 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:15.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:15.862 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:15.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:15.862 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:15.524+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:15.862 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:15.524+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:16.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:15.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:16.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:15.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:16.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:15.860+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:16.572 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:16.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:16.572 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:16.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:16.572 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:16.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:16.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:16.570+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:16.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:16.570+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:16.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:16.570+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:16.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:16.570+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:16.846+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:17.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:16.846+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:17.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:16.846+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:17.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:17.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:17.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:17.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:17.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:17.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:17.640 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:17.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:17.640 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:17.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:17.640 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:17.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:17.640 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:17.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:17.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:17.637+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50201 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:17.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:17.857+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:17.857+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:17.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:17.857+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:18.331+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:18.331+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:18.331+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:18.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:18.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:18.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:18.489+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:18.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:18.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:19.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:18.827+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:19.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:19.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:19.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:19.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:19.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:19.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:19.529+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:19.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:19.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:20.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:19.813+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:20.553 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:20.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:20.553 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:20.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:20.553 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:20.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:20.809 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:20.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:20.809 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:20.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:20.809 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:20.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:20.809 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:20.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:21.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:20.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:21.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:20.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:21.228 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:20.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:21.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:21.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:21.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:21.788+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:21.788+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:21.788+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:21.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:21.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:21.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:21.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:22.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:22.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:22.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:22.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:22.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:22.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:22.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:22.645+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50275 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:22.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:22.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:22.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:22.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:22.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:22.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:22.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:22.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:22.581+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:23.560 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:23.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:23.560 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:23.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:23.560 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:23.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:23.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:23.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:23.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:23.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:23.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:23.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:23.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:23.558+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:24.572 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:24.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:24.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:24.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:24.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:24.242+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:24.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:24.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:24.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:24.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:24.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:24.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:24.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:24.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:25.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:25.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:25.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:25.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:25.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:25.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:25.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:25.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:25.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:25.590+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:25.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:25.590+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:26.155+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:26.155+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:26.155+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:26.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:26.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:26.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:26.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:26.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:26.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:26.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:26.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:26.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:27.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:27.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:27.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:27.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:27.653+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50347 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:27.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:27.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:27.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:27.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:27.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:27.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:27.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:27.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:27.582+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:28.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:28.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:28.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:28.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:28.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:28.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:28.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:28.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:28.619+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:28.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:28.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:28.595+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:28.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:28.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:29.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:29.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:29.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:29.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:29.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:29.576+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:29.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:29.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:29.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:29.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:29.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:30.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:30.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:30.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:30.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:30.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:30.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:30.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:30.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:30.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:30.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:30.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:30.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:30.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:30.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:30.565+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:31.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:31.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:31.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:31.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:31.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:31.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:31.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:31.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:31.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:31.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:31.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:32.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:32.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:32.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:32.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:32.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:32.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:32.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:32.659+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50421 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:32.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:32.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:32.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:32.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:32.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:32.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:32.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:32.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:32.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:33.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:33.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:33.163+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:33.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:33.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:33.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:33.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:33.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:33.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:33.794 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:33.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:33.794 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:33.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:33.794 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:33.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:33.794 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:33.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:34.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:34.126+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:34.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:34.126+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:34.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:34.126+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:34.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:34.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:34.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:34.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:34.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:34.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:34.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:34.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:35.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:35.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:35.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:35.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:35.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:35.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:35.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:35.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:35.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:35.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:35.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:35.627+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:36.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:36.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:36.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:36.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:36.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:36.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:36.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:36.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:36.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:36.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:36.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:36.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:36.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:36.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:36.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:36.815 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:36.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:37.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:37.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:37.073+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:37.792 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:37.667+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50493 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:37.792 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:37.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:37.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:37.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:37.793 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:37.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:37.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:37.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:37.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:37.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:37.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:37.646+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:37.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:37.646+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:38.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:38.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:38.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:38.093+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:38.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:38.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:38.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:38.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:38.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:38.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:38.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:39.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:39.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:39.131+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:39.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:39.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:39.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:39.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:39.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:39.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:39.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:39.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:39.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:39.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:40.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:40.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:40.114+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:40.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:40.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:40.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:40.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:40.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:40.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:40.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:40.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:40.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:40.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:41.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:41.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:41.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:41.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:41.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:41.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:41.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:41.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:41.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:41.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:41.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:41.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:41.635+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:42.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:42.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:42.146+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:42.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:42.675+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50567 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:42.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:42.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:42.615+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:42.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:42.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:42.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:42.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:42.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:42.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:43.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:43.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:43.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:43.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:43.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:43.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:43.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:43.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:43.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:43.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:43.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:44.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:44.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:44.161+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:44.564+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:44.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:44.564+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:44.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:44.564+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:44.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:44.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:44.628+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:44.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:44.628+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:45.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:45.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:45.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:45.613+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:45.613+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:45.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:45.613+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:45.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:45.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:45.611+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:45.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:45.611+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:46.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:46.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:46.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:46.123+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:46.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:46.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:46.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:46.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:46.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:46.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:46.585+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:46.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:46.585+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:47.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:47.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:47.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:47.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:47.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:47.683+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50639 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:47.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:47.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:47.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:47.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:47.593+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:47.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:47.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:47.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:47.627+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:47.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:47.627+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:48.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:48.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:48.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:48.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:48.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:48.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:48.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:48.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:48.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:48.586+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:48.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:48.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:48.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:48.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:49.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:48.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:49.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:49.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:49.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:49.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:49.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:49.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:49.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:49.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:49.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:49.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:49.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:49.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:50.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:49.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:50.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:50.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:50.604+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:50.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:50.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:50.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:50.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:50.561+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:50.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:50.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:50.980+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:51.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:51.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:51.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:51.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:51.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:51.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:51.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:51.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:52.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:51.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:52.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:51.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:52.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:51.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:52.959 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:52.691+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50713 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:52.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:52.575+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:52.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:52.575+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:52.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:52.575+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:52.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:52.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:52.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:52.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:52.960 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:52.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:52.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:52.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:52.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:52.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:52.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:53.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:53.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:53.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:53.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:53.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:53.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:53.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:53.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:53.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:53.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:53.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:53.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:53.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:53.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:53.953+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:53.953+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:53.953+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:54.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:54.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:54.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:54.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:54.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:54.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:54.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:54.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:54.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:54.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:54.607+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:54.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:54.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:54.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:55.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:55.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:55.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:55.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:55.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:55.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:55.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:55.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:55.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:55.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:55.950+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:55.950+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:56.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:55.950+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:56.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:56.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:56.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:56.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:56.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:56.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:56.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:56.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:56.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:56.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:56.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:56.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:56.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:56.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:56.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:56.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:57.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:56.977+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:57.822 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:12:57.698+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50785 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:12:57.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:57.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:57.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:57.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:57.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:57.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:57.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:57.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:57.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:57.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:57.824 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:57.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:57.824 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:57.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:57.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:57.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:57.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:58.546 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:58.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:58.546 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:58.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:58.546 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:58.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:58.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:58.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:58.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:58.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:58.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:58.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:58.936 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:58.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:58.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:58.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:59.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:58.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:12:59.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:59.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:59.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:59.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:59.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:12:59.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:12:59.974 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:59.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:59.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:59.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:59.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:59.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:12:59.975 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:12:59.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:59.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:59.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:00.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:12:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:12:59.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:00.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:00.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:00.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:00.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:00.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:00.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:00.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:00.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:00.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:00.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:00.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:00.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:00.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:00.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:00.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:00.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:00.945+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:01.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:01.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:01.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:01.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:01.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:01.677+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:01.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:01.677+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:01.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:01.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:01.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:01.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:01.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:01.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:01.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:01.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:01.958+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:02.919 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:02.707+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50859 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:02.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:02.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:02.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:02.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:02.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:02.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:02.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:02.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:02.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:02.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:02.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:03.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:03.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:03.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:03.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:03.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:03.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:03.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:03.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:03.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:03.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:03.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:03.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:03.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:03.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:03.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:03.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:03.949+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:04.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:04.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:04.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:04.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:04.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:04.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:04.915 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:04.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:04.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:04.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:04.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:04.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:04.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:04.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:04.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:04.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:05.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:04.913+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:05.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:05.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:05.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:05.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:05.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:05.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:05.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:05.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:05.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:05.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:05.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:05.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:05.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:05.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:05.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:05.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:06.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:05.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:06.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:06.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:06.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:06.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:06.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:06.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:06.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:06.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:06.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:06.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:07.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:06.899+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:07.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:07.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:07.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:07.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:07.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:07.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:07.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:07.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:07.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:07.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:07.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:07.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:07.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:07.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:07.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:07.716+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 50931 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:07.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:07.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:07.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:07.874+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:08.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:08.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:08.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:08.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:08.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:08.639+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:08.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:08.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:08.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:09.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:08.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:09.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:09.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:09.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:09.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:09.673 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:09.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:09.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:09.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:09.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:09.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:09.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:09.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:09.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:09.672+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:10.701 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:10.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:10.701 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:10.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:10.702 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:10.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:10.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:10.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:10.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:10.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:10.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:10.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:10.700+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:10.700+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:11.677 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:11.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:11.677 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:11.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:11.677 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:11.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:11.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:11.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:11.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:11.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:11.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:11.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:11.676+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:11.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:12.654 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:12.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:12.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:12.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:12.655 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:12.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:12.978 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:12.723+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51005 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:12.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:12.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:12.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:12.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:12.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:12.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:12.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:12.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:12.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:13.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:13.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:13.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:13.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:13.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:13.607+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:13.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:13.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:13.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:14.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:13.910+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:14.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:14.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:14.355+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:14.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:14.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:14.559+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:14.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:14.559+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:14.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:14.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:15.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:14.882+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:15.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:15.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:15.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:15.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:15.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:15.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:15.558+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:15.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:15.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:16.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:15.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:16.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:16.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:16.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:16.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:16.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:16.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:16.565+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:16.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:16.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:17.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:16.839+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:17.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:17.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:17.382+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:17.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:17.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:17.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:17.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:18.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:17.730+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51077 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:18.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:17.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:17.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:18.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:17.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:18.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:18.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:18.432+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:18.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:18.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:18.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:18.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:18.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:18.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:19.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:18.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:19.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:19.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:19.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:19.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:19.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:19.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:19.572+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:19.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:19.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:20.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:19.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:20.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:20.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:20.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:20.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:20.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:20.548+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:20.548+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:20.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:20.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:21.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:20.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:21.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:21.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:21.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:21.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:21.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:21.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:21.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:21.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:21.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:21.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:21.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:21.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:21.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:21.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:21.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:21.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:22.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:21.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:22.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:22.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:22.461+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:22.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:22.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:22.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:22.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:23.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:22.738+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51151 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:23.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:22.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:22.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:23.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:22.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:23.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:23.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:23.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:23.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:23.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:23.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:23.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:23.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:23.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:23.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:23.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:23.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:23.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:23.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:24.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:23.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:24.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:23.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:24.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:23.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:24.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:24.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:24.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:24.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:24.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:24.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:24.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:24.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:24.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:24.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:24.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:24.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:24.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:25.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:25.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:25.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:25.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:25.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:25.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:25.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:25.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:25.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:25.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:25.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:25.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:25.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:25.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:25.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:25.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:25.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:26.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:26.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:26.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:26.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:26.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:26.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:26.671+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:26.671+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:26.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:26.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:26.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:27.768 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:27.745+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51223 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:27.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:27.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:27.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:27.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:27.768 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:27.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:27.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:27.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:27.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:27.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:27.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:27.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:27.769 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:27.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:27.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:27.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:28.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:27.920+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:28.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:28.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:28.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:28.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:28.659+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:28.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:28.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:28.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:28.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:28.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:28.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:28.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:29.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:29.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:29.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:29.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:29.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:29.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:29.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:29.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:29.466+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:29.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:29.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:29.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:30.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:30.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:30.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:30.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:30.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:30.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:30.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:30.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:30.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:30.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:30.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:30.608+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:30.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:30.608+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:30.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:30.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:30.876+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:31.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:31.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:31.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:31.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:31.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:31.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:31.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:31.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:31.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:31.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:31.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:32.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:31.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:32.755 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:32.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:32.756 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:32.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:32.756 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:32.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:32.756 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:32.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:32.756 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:32.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:32.756 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:32.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:32.757 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:32.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:33.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:32.753+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51297 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:32.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:32.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:32.870+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:33.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:33.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:33.797 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:33.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:33.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:33.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:33.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:33.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:33.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:33.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:33.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:33.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:33.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:33.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:33.888+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:33.888+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:33.888+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:34.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:34.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:34.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:34.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:34.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:34.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:34.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:34.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:34.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:34.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:34.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:34.652+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:34.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:34.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:35.231 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:34.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:35.231 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:34.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:35.231 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:34.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:35.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:35.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:35.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:35.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:35.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:35.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:35.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:35.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:35.635+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:35.833 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:35.635+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:35.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:35.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:35.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:36.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:36.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:36.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:36.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:36.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:36.589+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:36.589+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:36.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:37.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:36.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:37.247 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:36.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:37.763 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:37.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:37.764 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:37.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:37.764 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:37.508+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:37.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:37.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:37.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:37.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:37.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:37.574+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:37.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:37.574+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:38.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:37.761+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51369 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:37.900+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:37.900+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:37.900+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:38.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:38.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:38.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:38.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:38.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:38.528+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:38.528+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:38.933+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:38.933+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:39.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:38.933+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:39.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:39.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:39.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:39.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:39.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:39.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:39.571+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:39.571+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:39.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:39.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:39.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:40.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:40.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:40.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:40.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:40.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:40.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:40.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:40.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:40.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:40.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:41.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:40.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:41.900 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:41.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:41.900 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:41.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:41.900 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:41.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:41.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:41.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:41.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:41.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:41.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:41.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:41.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:41.520+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:41.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:41.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:41.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:42.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:42.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:42.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:42.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:42.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:42.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:42.772 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:42.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:42.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:42.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:42.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:42.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:42.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:42.497+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:43.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:42.769+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51442 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:42.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:42.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:42.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:43.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:43.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:43.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:43.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:43.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:43.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:43.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:43.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:43.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:43.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:43.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:43.543+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:43.799 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:43.543+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:43.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:43.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:43.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:44.917 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:44.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:44.917 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:44.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:44.917 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:44.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:44.917 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:44.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:44.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:44.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:44.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:44.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:44.918 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:44.498+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:44.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:44.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:45.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:44.915+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:45.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:45.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:45.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:45.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:45.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:45.513+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:45.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:45.513+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:45.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:45.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:45.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:45.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:45.870 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:45.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:45.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:45.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:46.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:45.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:46.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:46.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:46.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:46.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:46.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:46.506+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:46.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:46.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:46.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:46.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:46.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:46.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:46.478+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:46.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:46.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:46.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:47.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:47.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:47.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:47.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:47.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:47.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:47.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:47.493+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:47.780 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:47.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:47.780 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:47.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:47.780 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:47.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:48.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:47.777+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51514 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:47.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:47.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:48.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:47.886+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:48.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:48.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:48.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:48.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:48.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:48.458+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:48.458+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:48.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:48.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:48.852+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:49.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:49.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:49.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:49.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:49.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:49.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:49.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:49.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:49.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:49.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:49.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:49.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:49.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:49.477+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:50.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:49.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:50.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:49.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:50.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:49.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:50.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:50.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:50.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:50.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:50.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:50.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:50.500+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:50.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:51.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:50.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:51.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:50.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:51.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:51.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:51.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:51.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:51.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:51.475+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:51.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:51.475+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:51.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:51.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:52.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:51.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:52.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:52.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:52.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:52.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:52.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:52.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:52.479+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:53.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:52.786+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51588 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:52.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:52.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:53.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:52.836+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:53.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:53.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:53.800 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:53.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:53.800 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:53.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:53.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:53.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:53.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:53.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:53.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:53.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:53.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:53.504+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:53.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:53.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:54.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:53.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:54.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:54.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:54.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:54.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:54.805 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:54.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:54.805 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:54.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:54.806 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:54.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:54.806 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:54.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:54.806 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:54.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:55.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:54.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:55.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:54.803+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:55.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:54.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:55.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:55.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:55.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:55.552+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:55.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:55.552+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:55.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:55.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:55.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:55.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:55.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:55.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:55.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:55.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:55.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:55.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:56.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:56.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:56.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:56.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:56.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:56.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:56.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:56.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:56.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:56.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:56.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:56.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:56.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:56.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:56.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:56.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:57.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:56.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:57.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:57.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:57.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:57.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:57.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:57.531+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:57.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:57.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:57.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:57.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:57.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:57.612+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:57.796 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:57.612+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:58.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:13:57.794+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51660 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:13:58.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:57.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:57.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:58.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:57.841+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:58.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:58.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:58.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:58.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:58.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:58.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:58.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:58.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:58.807+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:13:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:58.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:58.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:58.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:58.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:58.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:58.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:59.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:59.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:59.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:59.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:59.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:13:59.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:13:59.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:59.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:59.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:59.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:59.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:59.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:13:59.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:13:59.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:00.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:59.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:00.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:59.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:00.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:13:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:13:59.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:00.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:00.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:00.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:00.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:00.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:00.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:00.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:00.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:00.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:00.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:00.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:00.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:00.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:00.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:00.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:01.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:00.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:01.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:00.811+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:01.829 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:01.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:01.829 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:01.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:01.829 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:01.523+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:01.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:01.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:01.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:01.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:01.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:01.667+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:01.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:01.667+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:01.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:01.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:02.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:01.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:02.809 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:02.802+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51734 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:02.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:02.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:02.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:02.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:02.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:02.558+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:02.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:02.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:02.691+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:02.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:02.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:03.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:02.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:03.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:02.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:03.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:03.580+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:03.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:03.580+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:03.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:03.580+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:03.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:03.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:03.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:03.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:03.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:03.710+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:03.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:03.710+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:03.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:03.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:04.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:03.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:04.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:04.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:04.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:04.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:04.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:05.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:04.856+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:05.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:04.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:04.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:04.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:05.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:04.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:05.674+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:05.674+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:05.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:05.674+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:05.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:05.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:05.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:05.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:05.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:05.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:05.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:05.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:05.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:05.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:06.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:06.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:06.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:06.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:06.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:06.648+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:06.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:06.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:06.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:06.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:06.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:06.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:06.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:06.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:06.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:06.735+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:06.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:06.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:07.934 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:07.809+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51806 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:07.981 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:07.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:07.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:07.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:07.982 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:07.668+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:07.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:07.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:07.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:07.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:07.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:07.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:07.983 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:07.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:07.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:07.780+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:07.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:07.780+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:07.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:07.780+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:08.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:08.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:08.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:08.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:08.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:08.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:08.781+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:08.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:08.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:08.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:08.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:08.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:09.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:09.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:09.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:09.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:09.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:09.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:09.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:09.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:09.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:09.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:09.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:09.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:09.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:09.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:10.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:10.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:10.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:10.702+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:10.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:10.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:10.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:10.800+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:10.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:10.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:10.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:10.693+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:10.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:10.693+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:11.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:11.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:11.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:11.756+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:11.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:11.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:11.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:11.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:11.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:11.656+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:11.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:11.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:11.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:11.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:11.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:11.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:12.940 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:12.816+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51880 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:12.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:12.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:12.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:12.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:12.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:12.664+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:12.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:12.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:12.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:12.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:12.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:12.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:12.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:12.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:12.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:12.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:12.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:12.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:12.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:12.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:13.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:13.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:13.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:13.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:13.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:13.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:13.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:13.680+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:13.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:13.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:13.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:13.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:13.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:13.816+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:14.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:14.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:14.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:14.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:14.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:14.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:14.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:14.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:14.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:14.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:14.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:14.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:14.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:14.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:14.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:14.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:15.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:15.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:15.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:15.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:15.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:15.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:15.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:15.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:15.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:15.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:15.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:15.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:15.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:15.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:16.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:16.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:16.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:16.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:16.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:16.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:16.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:16.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:16.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:16.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:16.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:16.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:16.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:16.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:16.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:16.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:17.947 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:17.824+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 51952 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:17.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:17.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:17.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:17.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:17.948 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:17.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:17.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:17.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:17.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:17.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:17.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:17.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:17.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:17.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:17.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:17.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:17.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:17.631+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:17.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:17.631+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:18.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:18.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:18.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:18.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:18.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:18.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:18.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:18.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:18.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:18.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:18.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:18.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:18.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:18.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:18.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:18.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:19.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:19.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:19.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:19.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:19.878+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:19.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:19.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:19.980 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:19.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:19.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:19.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:19.981 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:19.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:19.982 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:19.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:20.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:20.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:20.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:20.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:20.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:20.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:20.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:20.630+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:20.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:20.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:20.663+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:20.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:20.663+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:21.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:21.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:21.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:21.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:21.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:21.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:21.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:21.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:21.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:21.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:21.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:21.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:22.957 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:22.831+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52026 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:22.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:22.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:22.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:22.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:22.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:22.610+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:22.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:22.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:22.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:22.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:22.958 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:22.848+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:22.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:22.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:22.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:22.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:22.958 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:22.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:22.959 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:22.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:23.863 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:23.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:23.864 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:23.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:23.864 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:23.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:23.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:23.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:23.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:23.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:23.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:23.640+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:23.864 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:23.640+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:23.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:23.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:24.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:23.862+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:24.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:24.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:24.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:24.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:24.842 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:24.573+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:24.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:24.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:24.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:24.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:24.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:24.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:24.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:25.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:24.840+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:25.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:25.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:25.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:25.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:25.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:25.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:25.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:25.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:26.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:25.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:26.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:25.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:26.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:26.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:26.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:26.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:26.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:26.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:26.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:26.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:26.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:26.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:27.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:26.911+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:27.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:27.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:27.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:27.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:27.841 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:27.527+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:27.841 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:27.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:27.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:27.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:27.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:27.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:27.842 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:27.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:28.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:27.838+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52098 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:27.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:27.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:28.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:27.934+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:28.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:28.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:28.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:28.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:28.934 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:28.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:28.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:28.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:28.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:28.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:28.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:28.699+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:28.935 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:28.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:28.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:28.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:29.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:28.932+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:29.932 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:29.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:29.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:29.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:29.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:29.540+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:29.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:29.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:29.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:29.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:29.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:29.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:29.933 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:29.651+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:29.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:29.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:30.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:29.930+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:30.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:30.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:30.526+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:30.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:30.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:30.651+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:30.652+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:30.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:30.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:31.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:30.975+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:31.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:31.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:31.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:31.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:31.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:31.647+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:31.647+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:31.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:32.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:31.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:32.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:31.994+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:32.848 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:32.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:32.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:32.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:32.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:32.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:32.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:32.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:32.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:32.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:32.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:32.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:32.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:32.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:33.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:32.846+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52172 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:33.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:32.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:32.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:33.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:32.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:33.802 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:33.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:33.802 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:33.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:33.802 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:33.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:33.802 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:33.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:33.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:33.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:33.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:33.601+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:33.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:33.601+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:33.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:33.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:34.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:33.938+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:34.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:34.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:34.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:34.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:34.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:34.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:34.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:34.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:34.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:34.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:34.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:34.984+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:35.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:35.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:35.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:35.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:35.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:35.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:35.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:35.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:35.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:35.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:35.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:35.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:35.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:35.532+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:35.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:35.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:36.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:35.941+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:36.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:36.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:36.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:36.566+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:36.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:36.567+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:36.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:36.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:36.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:36.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:36.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:36.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:36.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:36.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:36.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:36.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:37.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:36.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:37.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:37.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:37.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:37.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:37.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:37.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:37.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:37.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:37.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:37.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:37.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:37.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:37.858 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:37.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:38.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:37.854+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52244 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:38.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:37.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:37.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:38.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:37.937+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:38.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:38.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:38.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:38.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:38.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:38.578+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:38.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:38.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:38.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:38.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:38.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:38.576+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:38.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:38.576+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:38.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:38.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:39.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:38.929+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:39.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:39.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:39.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:39.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:39.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:39.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:39.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:39.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:39.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:39.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:39.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:39.550+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:39.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:39.551+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:39.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:40.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:39.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:40.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:39.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:40.861 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:40.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:40.861 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:40.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:40.861 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:40.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:40.861 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:40.516+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:40.862 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:40.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:40.862 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:40.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:40.862 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:40.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:40.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:40.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:41.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:40.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:41.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:41.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:41.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:41.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:41.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:41.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:41.506+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:41.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:41.506+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:42.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:41.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:41.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:42.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:41.837+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:42.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:42.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:42.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:42.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:42.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:42.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:42.460+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:42.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:42.460+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:43.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:42.863+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52317 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:43.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:42.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:42.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:43.229 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:42.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:43.802 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:43.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:43.802 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:43.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:43.802 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:43.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:43.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:43.490+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:43.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:43.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:43.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:43.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:43.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:43.539+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:43.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:43.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:44.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:43.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:44.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:44.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:44.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:44.521+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:44.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:44.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:44.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:44.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:44.491+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:45.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:44.879+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.879840+0000) 2024-02-19T20:14:45.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:44.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:45.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:44.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:45.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:44.810+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:45.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:44.862+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.862882+0000) 2024-02-19T20:14:45.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:45.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:45.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:45.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:45.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:45.549+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:45.470+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:45.470+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:45.470+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:45.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:45.470+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:46.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:45.881+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.881504+0000) 2024-02-19T20:14:46.085 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:45.771+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.771680+0000) 2024-02-19T20:14:46.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:45.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:46.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:45.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:46.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:45.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:46.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:45.862+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.863057+0000) 2024-02-19T20:14:46.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:46.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:46.773 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:46.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:46.773 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:46.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:46.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:46.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:46.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:46.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:46.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:46.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:46.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:46.488+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:47.085 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:46.771+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.771806+0000) 2024-02-19T20:14:47.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:46.881+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.881744+0000) 2024-02-19T20:14:47.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:46.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:47.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:46.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:47.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:46.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:47.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:46.862+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.863207+0000) 2024-02-19T20:14:47.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:47.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:47.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:47.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:47.773 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:47.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:47.773 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:47.771+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.771934+0000) 2024-02-19T20:14:47.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:47.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:47.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:47.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:47.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:47.487+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:47.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:47.488+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:48.085 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:47.870+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52390 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:48.086 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:47.866+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.866487+0000) 2024-02-19T20:14:48.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:47.868+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.869072+0000) 2024-02-19T20:14:48.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:47.881+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.881949+0000) 2024-02-19T20:14:48.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:47.863+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.863366+0000) 2024-02-19T20:14:48.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:47.864+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.865379+0000) 2024-02-19T20:14:48.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:47.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:48.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:47.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:48.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:47.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:48.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:48.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:48.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:48.453+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:48.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:48.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:48.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:48.476+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:49.085 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:48.771+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.772090+0000) 2024-02-19T20:14:49.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:48.884+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.884868+0000) 2024-02-19T20:14:49.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:48.863+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.863576+0000) 2024-02-19T20:14:49.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:48.914+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:49.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:48.914+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:49.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:48.914+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:49.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:49.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:49.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:49.408+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:49.469+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:49.469+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:49.469+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:49.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:49.469+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:50.086 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:49.771+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.772212+0000) 2024-02-19T20:14:50.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:49.887+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.887492+0000) 2024-02-19T20:14:50.086 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:49.863+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.863693+0000) 2024-02-19T20:14:50.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:49.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:50.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:49.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:50.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:49.891+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:50.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:50.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:50.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:50.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:50.456+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:50.456+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:50.456+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:50.456+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:51.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:50.888+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.889204+0000) 2024-02-19T20:14:51.086 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:50.771+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.772405+0000) 2024-02-19T20:14:51.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:50.863+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.863882+0000) 2024-02-19T20:14:51.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:50.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:51.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:50.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:51.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:50.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:51.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:51.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:51.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:51.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:51.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:51.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:51.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:51.453+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:51.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:51.453+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:52.086 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:51.772+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.772636+0000) 2024-02-19T20:14:52.087 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:51.889+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.889514+0000) 2024-02-19T20:14:52.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:51.863+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.864063+0000) 2024-02-19T20:14:52.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:51.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:52.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:51.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:52.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:51.898+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:52.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:52.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:52.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:52.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:52.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:52.493+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:52.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:52.493+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:53.086 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:52.878+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52463 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:53.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:52.772+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.772808+0000) 2024-02-19T20:14:53.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:52.872+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.872930+0000) 2024-02-19T20:14:53.087 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:52.875+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.875499+0000) 2024-02-19T20:14:53.088 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:52.889+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.889751+0000) 2024-02-19T20:14:53.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:52.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.864235+0000) 2024-02-19T20:14:53.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:52.872+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.873014+0000) 2024-02-19T20:14:53.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:52.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:53.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:52.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:53.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:52.893+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:53.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:53.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:53.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:53.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:53.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:53.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:53.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:53.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:53.455+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:54.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:53.772+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.772975+0000) 2024-02-19T20:14:54.087 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:53.893+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.893391+0000) 2024-02-19T20:14:54.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:53.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.864339+0000) 2024-02-19T20:14:54.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:53.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:54.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:53.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:54.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:53.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:54.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:54.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:54.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:54.395+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:54.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:54.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:54.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:54.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:54.491+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:55.086 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:54.772+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.773129+0000) 2024-02-19T20:14:55.087 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:54.895+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.896074+0000) 2024-02-19T20:14:55.087 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:54.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.864497+0000) 2024-02-19T20:14:55.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:54.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:55.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:54.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:55.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:54.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:55.645 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:55.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:55.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:55.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:55.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:55.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:55.646 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:55.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:55.646 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:55.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:55.646 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:55.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:55.646 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:55.482+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:55.898 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:55.772+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.773296+0000) 2024-02-19T20:14:55.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:55.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.864720+0000) 2024-02-19T20:14:56.158 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:55.897+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.897762+0000) 2024-02-19T20:14:56.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:55.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:56.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:55.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:56.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:55.983+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:56.439 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:56.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:56.440 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:56.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:56.440 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:56.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:56.438+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:56.438+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:56.438+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:56.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:56.439+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:57.091 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:56.897+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.897963+0000) 2024-02-19T20:14:57.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:56.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.773492+0000) 2024-02-19T20:14:57.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:56.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.864902+0000) 2024-02-19T20:14:57.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:56.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:57.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:56.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:57.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:56.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:57.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:57.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:57.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:57.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:57.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:57.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:57.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:57.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:57.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:57.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:57.481+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:58.091 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:14:57.886+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52536 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:14:58.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:57.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.773679+0000) 2024-02-19T20:14:58.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:57.880+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.881187+0000) 2024-02-19T20:14:58.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:57.883+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.884174+0000) 2024-02-19T20:14:58.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:57.897+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.898181+0000) 2024-02-19T20:14:58.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:57.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.865041+0000) 2024-02-19T20:14:58.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:57.880+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.880868+0000) 2024-02-19T20:14:58.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:57.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:58.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:57.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:58.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:57.896+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:58.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:58.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:58.357+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:58.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:58.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:58.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:58.494+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:59.091 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:58.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.773894+0000) 2024-02-19T20:14:59.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:58.901+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.901190+0000) 2024-02-19T20:14:59.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:58.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.865240+0000) 2024-02-19T20:14:59.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:58.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:59.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:58.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:59.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:58.881+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:14:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:59.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:59.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:59.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:59.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:14:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:59.463+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:59.463+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:59.463+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:14:59.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:59.463+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:00.091 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:14:59.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.774066+0000) 2024-02-19T20:15:00.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:14:59.903+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.903946+0000) 2024-02-19T20:15:00.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:59.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.865413+0000) 2024-02-19T20:15:00.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:59.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:00.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:59.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:00.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:14:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:14:59.877+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:00.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:00.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:00.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:00.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:00.688 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:00.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:00.688 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:00.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:00.689 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:00.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:00.689 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:00.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:00.689 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:00.459+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:00.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:00.905+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.905767+0000) 2024-02-19T20:15:00.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:00.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.774274+0000) 2024-02-19T20:15:00.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:00.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:00.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:00.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:00.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:00.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:00.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:00.865+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.865660+0000) 2024-02-19T20:15:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:01.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:01.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:01.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:01.418+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:01.418+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:01.418+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:01.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:01.419+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:02.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:01.772+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.774424+0000) 2024-02-19T20:15:02.092 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:01.904+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.906143+0000) 2024-02-19T20:15:02.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:01.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:02.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:01.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:02.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:01.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:02.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:01.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.865808+0000) 2024-02-19T20:15:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:02.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:02.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:02.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:02.371+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:02.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:02.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:02.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:02.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:02.449+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:03.092 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:02.893+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52609 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:03.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:02.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.774543+0000) 2024-02-19T20:15:03.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:02.887+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.889302+0000) 2024-02-19T20:15:03.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:02.891+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.892484+0000) 2024-02-19T20:15:03.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:02.905+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.906469+0000) 2024-02-19T20:15:03.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:02.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:03.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:02.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:03.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:02.778+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:03.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:02.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.865916+0000) 2024-02-19T20:15:03.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:02.887+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.888844+0000) 2024-02-19T20:15:03.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:03.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:03.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:03.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:03.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:03.330+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:03.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:03.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:03.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:03.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:03.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:04.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:03.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.774722+0000) 2024-02-19T20:15:04.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:03.908+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.909068+0000) 2024-02-19T20:15:04.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:03.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:04.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:03.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:04.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:03.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:04.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:03.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.866075+0000) 2024-02-19T20:15:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:04.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:04.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:04.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:04.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:04.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:04.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:04.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:04.452+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:05.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:04.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.774880+0000) 2024-02-19T20:15:05.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:04.910+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.911706+0000) 2024-02-19T20:15:05.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:04.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:05.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:04.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:05.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:04.763+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:05.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:04.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.866171+0000) 2024-02-19T20:15:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:05.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:05.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:05.366+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:05.428+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:05.428+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:05.428+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:05.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:05.428+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:06.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:05.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.775077+0000) 2024-02-19T20:15:06.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:05.912+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.913382+0000) 2024-02-19T20:15:06.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:05.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:06.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:05.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:06.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:05.731+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:06.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:05.864+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.866350+0000) 2024-02-19T20:15:06.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:06.407+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:06.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:06.407+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:06.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:06.407+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:06.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:06.407+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:06.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:06.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:06.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:07.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:06.912+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.913749+0000) 2024-02-19T20:15:07.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:06.773+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.775285+0000) 2024-02-19T20:15:07.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:06.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:07.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:06.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:07.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:06.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:07.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:06.865+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.866530+0000) 2024-02-19T20:15:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:07.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:07.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:07.387+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:07.387+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:07.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:07.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:07.362+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:08.024 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:07.900+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52682 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:08.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:07.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.775425+0000) 2024-02-19T20:15:08.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:07.894+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.896269+0000) 2024-02-19T20:15:08.025 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:07.897+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.899256+0000) 2024-02-19T20:15:08.025 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:07.912+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.914017+0000) 2024-02-19T20:15:08.026 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:08.026 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:08.026 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:07.758+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:08.026 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:07.865+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.866689+0000) 2024-02-19T20:15:08.026 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:07.894+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.896368+0000) 2024-02-19T20:15:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:08.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:08.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:08.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:08.349+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:08.349+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:08.349+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:08.349+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:09.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:08.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.775524+0000) 2024-02-19T20:15:09.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:08.915+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.916485+0000) 2024-02-19T20:15:09.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:08.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:09.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:08.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:09.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:08.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:09.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:08.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.866890+0000) 2024-02-19T20:15:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:09.366+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:09.366+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:09.366+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:09.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:09.366+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:09.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:09.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:09.370+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:10.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:09.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.775672+0000) 2024-02-19T20:15:10.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:09.918+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.919570+0000) 2024-02-19T20:15:10.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:09.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:10.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:09.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:10.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:09.838+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:10.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:09.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.867050+0000) 2024-02-19T20:15:10.353 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:10.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:10.353 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:10.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:10.353 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:10.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:10.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:10.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:10.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:10.351+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:11.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:10.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.775840+0000) 2024-02-19T20:15:11.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:10.920+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.921302+0000) 2024-02-19T20:15:11.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:10.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:11.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:10.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:11.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:10.843+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:11.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:10.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.867209+0000) 2024-02-19T20:15:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:11.342+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:11.342+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:11.342+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:11.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:11.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:11.364+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:11.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:11.365+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:12.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:11.775+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.776013+0000) 2024-02-19T20:15:12.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:11.920+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.921648+0000) 2024-02-19T20:15:12.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:11.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:12.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:11.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:12.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:11.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:12.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:11.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.867423+0000) 2024-02-19T20:15:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:12.345+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:12.345+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:12.345+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:12.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:12.345+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:12.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:12.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:12.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:12.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:13.029 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:12.908+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52755 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:13.030 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:12.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.776175+0000) 2024-02-19T20:15:13.030 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:12.902+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.904361+0000) 2024-02-19T20:15:13.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:12.904+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.906424+0000) 2024-02-19T20:15:13.031 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:12.920+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.921886+0000) 2024-02-19T20:15:13.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:12.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:13.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:12.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:13.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:12.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:13.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:12.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.867639+0000) 2024-02-19T20:15:13.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:12.901+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.903394+0000) 2024-02-19T20:15:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:13.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:13.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:13.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:13.320+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:13.311+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:13.311+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:13.311+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:13.311+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:14.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:13.774+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.776273+0000) 2024-02-19T20:15:14.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:13.924+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.925530+0000) 2024-02-19T20:15:14.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:13.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:14.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:13.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:14.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:13.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:14.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:13.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.867761+0000) 2024-02-19T20:15:14.350 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:14.301+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:14.350 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:14.301+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:14.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:14.301+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:14.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:14.301+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:14.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:14.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:14.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:15.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:14.775+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.776422+0000) 2024-02-19T20:15:15.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:14.927+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.928301+0000) 2024-02-19T20:15:15.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:14.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:15.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:14.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:15.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:14.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:15.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:14.866+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.867864+0000) 2024-02-19T20:15:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:15.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:15.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:15.340+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:15.340+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:15.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:15.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:15.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:16.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:15.775+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.776623+0000) 2024-02-19T20:15:16.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:15.929+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.930118+0000) 2024-02-19T20:15:16.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:15.867+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.868065+0000) 2024-02-19T20:15:16.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:15.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:16.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:15.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:16.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:15.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:16.385+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:16.385+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:16.385+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:16.385+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:16.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:16.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:16.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:16.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:17.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:16.776+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.776831+0000) 2024-02-19T20:15:17.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:16.929+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.930423+0000) 2024-02-19T20:15:17.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:16.867+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.868263+0000) 2024-02-19T20:15:17.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:16.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:17.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:16.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:17.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:16.889+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:17.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:17.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:17.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:17.392+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:17.392+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:17.392+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:17.392+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:18.038 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:17.915+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52828 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:18.039 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:17.775+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.776999+0000) 2024-02-19T20:15:18.039 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:17.909+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.910798+0000) 2024-02-19T20:15:18.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:17.913+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.914534+0000) 2024-02-19T20:15:18.039 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:17.929+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.930752+0000) 2024-02-19T20:15:18.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:17.867+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.868420+0000) 2024-02-19T20:15:18.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:17.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:18.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:17.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:18.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:17.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:18.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:17.909+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.910874+0000) 2024-02-19T20:15:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:18.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:18.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:18.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:18.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:18.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:18.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:18.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:18.370+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:19.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:18.775+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.777177+0000) 2024-02-19T20:15:19.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:18.932+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.933129+0000) 2024-02-19T20:15:19.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:18.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:19.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:18.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:19.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:18.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:19.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:18.867+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.868623+0000) 2024-02-19T20:15:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:19.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:19.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:19.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:19.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:19.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:19.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:19.401+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:20.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:19.776+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.777336+0000) 2024-02-19T20:15:20.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:19.934+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.935843+0000) 2024-02-19T20:15:20.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:19.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:20.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:19.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:20.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:19.830+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:20.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:19.868+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.868795+0000) 2024-02-19T20:15:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:20.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:20.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:20.346+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:20.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:20.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:20.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:20.401+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:21.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:20.776+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.777520+0000) 2024-02-19T20:15:21.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:20.936+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.937539+0000) 2024-02-19T20:15:21.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:20.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:21.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:20.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:21.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:20.797+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:21.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:20.867+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.868949+0000) 2024-02-19T20:15:21.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:21.313+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:21.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:21.313+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:21.423 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:21.313+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:21.422+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:21.422+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:21.422+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:21.422+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:22.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:21.936+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.937751+0000) 2024-02-19T20:15:22.096 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:21.777+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.777734+0000) 2024-02-19T20:15:22.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:21.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:22.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:21.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:22.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:21.817+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:22.096 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:21.868+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.869071+0000) 2024-02-19T20:15:22.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:22.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:22.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:22.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:22.435 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:22.328+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:22.434+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:22.434+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:22.434+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:22.434+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:23.048 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:22.922+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52901 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:23.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:22.776+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.777926+0000) 2024-02-19T20:15:23.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:22.917+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.919083+0000) 2024-02-19T20:15:23.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:22.919+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.920718+0000) 2024-02-19T20:15:23.049 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:22.936+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.937969+0000) 2024-02-19T20:15:23.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:22.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:23.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:22.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:23.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:22.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:23.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:22.868+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.869257+0000) 2024-02-19T20:15:23.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:22.916+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.918293+0000) 2024-02-19T20:15:23.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:23.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:23.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:23.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:23.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:23.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:23.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:23.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:23.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:23.442+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:24.096 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:23.776+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.778028+0000) 2024-02-19T20:15:24.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:23.939+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.940724+0000) 2024-02-19T20:15:24.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:23.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:24.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:23.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:24.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:23.771+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:24.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:23.868+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.869412+0000) 2024-02-19T20:15:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:24.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:24.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:24.387+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:24.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:24.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:24.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:24.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:24.489+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:25.096 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:24.777+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.778191+0000) 2024-02-19T20:15:25.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:24.942+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.943620+0000) 2024-02-19T20:15:25.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:24.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:25.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:24.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:25.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:24.742+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:25.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:24.869+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.869561+0000) 2024-02-19T20:15:25.449 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:25.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:25.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:25.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:25.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:25.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:25.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:25.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:25.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:25.448+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:26.096 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:25.777+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.778406+0000) 2024-02-19T20:15:26.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:25.944+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.945729+0000) 2024-02-19T20:15:26.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:25.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:26.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:25.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:26.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:25.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:26.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:25.869+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.869729+0000) 2024-02-19T20:15:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:26.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:26.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:26.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:52.250965+0000 (oldest deadline 2024-02-19T20:05:12.250965+0000) 2024-02-19T20:15:26.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:26.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:26.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:26.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:26.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:26.481+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:26.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:26.481+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:27.096 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:26.778+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.778649+0000) 2024-02-19T20:15:27.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:26.945+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.945948+0000) 2024-02-19T20:15:27.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:26.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:27.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:26.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:27.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:26.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:04:53.305133+0000 (oldest deadline 2024-02-19T20:05:13.305133+0000) 2024-02-19T20:15:27.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:26.869+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.869914+0000) 2024-02-19T20:15:27.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:27.483+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:27.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:27.483+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:27.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:27.483+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:27.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:27.483+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:28.054 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:27.930+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 52974 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:28.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:27.778+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.778826+0000) 2024-02-19T20:15:28.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:27.924+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.925610+0000) 2024-02-19T20:15:28.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:27.928+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.929530+0000) 2024-02-19T20:15:28.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:27.944+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.946144+0000) 2024-02-19T20:15:28.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:27.869+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.870094+0000) 2024-02-19T20:15:28.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:27.924+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.925543+0000) 2024-02-19T20:15:28.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:28.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:28.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:28.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:28.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:28.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:28.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:28.504+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:29.096 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:28.778+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.778986+0000) 2024-02-19T20:15:29.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:28.948+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.948589+0000) 2024-02-19T20:15:29.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:28.869+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.870295+0000) 2024-02-19T20:15:29.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:29.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:29.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:29.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:29.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:29.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:29.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:29.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:30.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:29.778+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.779159+0000) 2024-02-19T20:15:30.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:29.950+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.951214+0000) 2024-02-19T20:15:30.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:29.869+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.870409+0000) 2024-02-19T20:15:30.871 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:30.778+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.779342+0000) 2024-02-19T20:15:30.871 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:30.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:30.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:30.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:30.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:30.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:30.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:30.547+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:31.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:30.952+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.953038+0000) 2024-02-19T20:15:31.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:30.870+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.870598+0000) 2024-02-19T20:15:31.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:31.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:31.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:31.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:31.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:31.527+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:31.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:31.527+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:32.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:31.779+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.779553+0000) 2024-02-19T20:15:32.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:31.952+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.953271+0000) 2024-02-19T20:15:32.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:31.870+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.870793+0000) 2024-02-19T20:15:32.871 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:32.779+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.779758+0000) 2024-02-19T20:15:32.871 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:32.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:32.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:32.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:32.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:32.560+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:32.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:32.560+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:33.184 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:32.940+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53047 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:33.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:32.932+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.933284+0000) 2024-02-19T20:15:33.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:32.936+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.936479+0000) 2024-02-19T20:15:33.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:32.953+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.953465+0000) 2024-02-19T20:15:33.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:32.870+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.870918+0000) 2024-02-19T20:15:33.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:32.933+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.933708+0000) 2024-02-19T20:15:33.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:33.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:33.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:33.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:33.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:33.516+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:33.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:33.517+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:34.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:33.779+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.779923+0000) 2024-02-19T20:15:34.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:33.955+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.955937+0000) 2024-02-19T20:15:34.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:33.870+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.871101+0000) 2024-02-19T20:15:34.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:34.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:34.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:34.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:34.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:34.529+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:34.782 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:34.529+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:35.097 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:34.958+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.958575+0000) 2024-02-19T20:15:35.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:34.870+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.871254+0000) 2024-02-19T20:15:35.098 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:34.779+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.780037+0000) 2024-02-19T20:15:35.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:35.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:35.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:35.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:35.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:35.502+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:35.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:35.502+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:36.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:35.779+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.780225+0000) 2024-02-19T20:15:36.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:35.959+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.960308+0000) 2024-02-19T20:15:36.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:35.871+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.871405+0000) 2024-02-19T20:15:36.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:36.779+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.780417+0000) 2024-02-19T20:15:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:36.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:36.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:36.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:36.541+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:36.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:36.541+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:37.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:36.960+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.960540+0000) 2024-02-19T20:15:37.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:36.871+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.871533+0000) 2024-02-19T20:15:37.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:37.780+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.780553+0000) 2024-02-19T20:15:37.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:37.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:37.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:37.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:37.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:37.572+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:37.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:37.573+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:38.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:37.941+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.941878+0000) 2024-02-19T20:15:38.189 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:37.948+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53120 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:38.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:37.945+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.945589+0000) 2024-02-19T20:15:38.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:37.960+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.960782+0000) 2024-02-19T20:15:38.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:37.871+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.871663+0000) 2024-02-19T20:15:38.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:37.941+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.942267+0000) 2024-02-19T20:15:38.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:38.780+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.780750+0000) 2024-02-19T20:15:38.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:38.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:38.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:38.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:38.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:38.620+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:05:05.206803+0000 (oldest deadline 2024-02-19T20:05:25.206803+0000) 2024-02-19T20:15:38.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:38.620+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:39.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:38.963+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.963701+0000) 2024-02-19T20:15:39.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:38.871+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.871833+0000) 2024-02-19T20:15:39.937 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:39.780+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.780896+0000) 2024-02-19T20:15:39.937 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:39.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:39.938 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:39.871+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.871974+0000) 2024-02-19T20:15:40.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:39.966+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.966442+0000) 2024-02-19T20:15:40.937 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:40.871+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.872171+0000) 2024-02-19T20:15:40.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:40.780+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.781063+0000) 2024-02-19T20:15:40.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:40.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:41.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:40.968+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.968254+0000) 2024-02-19T20:15:41.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:41.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:41.938 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:41.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.872324+0000) 2024-02-19T20:15:41.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:41.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.781288+0000) 2024-02-19T20:15:42.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:41.968+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.968631+0000) 2024-02-19T20:15:42.938 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:42.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:42.938 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:42.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.872484+0000) 2024-02-19T20:15:42.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:42.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.781487+0000) 2024-02-19T20:15:43.191 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:42.950+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.950939+0000) 2024-02-19T20:15:43.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:42.950+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.951232+0000) 2024-02-19T20:15:43.192 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:42.957+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53194 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:43.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:42.953+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.954143+0000) 2024-02-19T20:15:43.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:42.968+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.968846+0000) 2024-02-19T20:15:43.938 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:43.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.872661+0000) 2024-02-19T20:15:43.938 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:43.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.781657+0000) 2024-02-19T20:15:43.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:43.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:44.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:43.971+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.971780+0000) 2024-02-19T20:15:44.975 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:44.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.781783+0000) 2024-02-19T20:15:44.976 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:44.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:44.976 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:44.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.872781+0000) 2024-02-19T20:15:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:44.974+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.974806+0000) 2024-02-19T20:15:46.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:45.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.872941+0000) 2024-02-19T20:15:46.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:45.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.781916+0000) 2024-02-19T20:15:46.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:45.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:46.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:45.976+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.976908+0000) 2024-02-19T20:15:46.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:46.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.782124+0000) 2024-02-19T20:15:46.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:46.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:46.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:46.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.873103+0000) 2024-02-19T20:15:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:46.977+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.977259+0000) 2024-02-19T20:15:47.960 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:47.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.873350+0000) 2024-02-19T20:15:47.960 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:47.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.782329+0000) 2024-02-19T20:15:47.961 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:47.699+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:48.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:47.959+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.959913+0000) 2024-02-19T20:15:48.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:47.965+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53267 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:47.963+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.963618+0000) 2024-02-19T20:15:48.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:47.977+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.977630+0000) 2024-02-19T20:15:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:47.959+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.960268+0000) 2024-02-19T20:15:48.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:48.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:48.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:48.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:48.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:48.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:48.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:48.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.782495+0000) 2024-02-19T20:15:48.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:48.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:48.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:48.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:48.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:48.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:48.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.873528+0000) 2024-02-19T20:15:49.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:48.979+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.980326+0000) 2024-02-19T20:15:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:49.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:49.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:49.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:49.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:49.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.782660+0000) 2024-02-19T20:15:49.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:49.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:49.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:49.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:49.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:49.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.873693+0000) 2024-02-19T20:15:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:49.982+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.982832+0000) 2024-02-19T20:15:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:50.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:50.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:50.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:50.125+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:50.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:50.717+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:50.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:50.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.782799+0000) 2024-02-19T20:15:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:50.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:50.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:50.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:50.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:50.872+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.873890+0000) 2024-02-19T20:15:51.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:50.983+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.984558+0000) 2024-02-19T20:15:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:51.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:51.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:51.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:51.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:51.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:51.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:51.716+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:51.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.783045+0000) 2024-02-19T20:15:51.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:51.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:51.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:51.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:51.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:51.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:51.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:51.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.874057+0000) 2024-02-19T20:15:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:51.983+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.984808+0000) 2024-02-19T20:15:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:52.100+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:52.100+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:52.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:52.100+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:52.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:52.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:52.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:52.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:52.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:52.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:52.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:52.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.874240+0000) 2024-02-19T20:15:52.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:52.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.783259+0000) 2024-02-19T20:15:52.970 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:52.691+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:53.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:52.966+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.967780+0000) 2024-02-19T20:15:53.228 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:52.972+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53340 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:52.970+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.971514+0000) 2024-02-19T20:15:53.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:52.983+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.985006+0000) 2024-02-19T20:15:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:52.966+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.967896+0000) 2024-02-19T20:15:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:53.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:53.229 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:53.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:53.230 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:53.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:53.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:53.781+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.783372+0000) 2024-02-19T20:15:53.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:53.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:53.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:53.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:53.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:53.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:53.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.874341+0000) 2024-02-19T20:15:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:53.986+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.988129+0000) 2024-02-19T20:15:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:54.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:54.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:54.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:54.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:54.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.783544+0000) 2024-02-19T20:15:54.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:54.724+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:54.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:54.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:54.812+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:54.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:54.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.874500+0000) 2024-02-19T20:15:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:54.989+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.990724+0000) 2024-02-19T20:15:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:55.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:55.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:55.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:55.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:55.993 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:55.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.783675+0000) 2024-02-19T20:15:55.993 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:55.739+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:55.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:55.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:55.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:55.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:55.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:55.783+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:55.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:55.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.874643+0000) 2024-02-19T20:15:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:55.991+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.992490+0000) 2024-02-19T20:15:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:56.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:56.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:56.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:56.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:57.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:56.782+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.783856+0000) 2024-02-19T20:15:57.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:57.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:57.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:57.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:57.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:56.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:57.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:56.991+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.992774+0000) 2024-02-19T20:15:57.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:56.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:57.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:56.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:57.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:56.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:57.069 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:56.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.874844+0000) 2024-02-19T20:15:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:57.065+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:57.993 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:15:57.980+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53413 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:15:57.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:57.783+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.784046+0000) 2024-02-19T20:15:57.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:57.974+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.975683+0000) 2024-02-19T20:15:57.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:57.735+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:57.994 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:57.978+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.980229+0000) 2024-02-19T20:15:57.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:57.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:57.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:57.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:57.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:57.832+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:57.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:57.874+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.875054+0000) 2024-02-19T20:15:57.995 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:57.975+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.976560+0000) 2024-02-19T20:15:58.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:57.991+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.992970+0000) 2024-02-19T20:15:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:58.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:58.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:58.074+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:58.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:58.783+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.784228+0000) 2024-02-19T20:15:58.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:58.718+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:58.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:58.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:58.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:58.869+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:58.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:58.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.875241+0000) 2024-02-19T20:15:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:58.995+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.996651+0000) 2024-02-19T20:15:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:59.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:59.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:59.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:59.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:15:59.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:15:59.783+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.784397+0000) 2024-02-19T20:15:59.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:59.688+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:59.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:59.873+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.875406+0000) 2024-02-19T20:15:59.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:59.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:59.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:59.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:15:59.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:15:59.916+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:00.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:15:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:15:59.997+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.999407+0000) 2024-02-19T20:16:00.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:00.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:00.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:00.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:00.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:00.089+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:00.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:00.783+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.784536+0000) 2024-02-19T20:16:00.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:00.874+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.875567+0000) 2024-02-19T20:16:00.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:00.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:00.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:00.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:00.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:00.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:00.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:00.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:01.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:01.000+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.001313+0000) 2024-02-19T20:16:01.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:00.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:01.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:00.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:01.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:00.944+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:01.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:01.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:01.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:01.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:01.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:01.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:01.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:01.783+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.784733+0000) 2024-02-19T20:16:01.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:01.874+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.875726+0000) 2024-02-19T20:16:01.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:01.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:01.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:01.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:01.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:01.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:01.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:01.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:02.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:02.000+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.001668+0000) 2024-02-19T20:16:02.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:02.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:02.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:02.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:02.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:02.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:01.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:01.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:01.978+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:02.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:02.784+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.784905+0000) 2024-02-19T20:16:02.942 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:02.875+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.875930+0000) 2024-02-19T20:16:02.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:02.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:02.942 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:02.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:02.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:02.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:02.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:02.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:03.195 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:02.988+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53486 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:03.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:02.984+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.986224+0000) 2024-02-19T20:16:03.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:03.000+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.001888+0000) 2024-02-19T20:16:03.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:02.982+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.983841+0000) 2024-02-19T20:16:03.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:03.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:03.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:03.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:03.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:03.014+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:03.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:02.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:03.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:02.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:03.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:02.942+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:03.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:02.981+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.983128+0000) 2024-02-19T20:16:03.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:03.784+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.785076+0000) 2024-02-19T20:16:03.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:03.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:03.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:03.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:03.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:03.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:03.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:03.594+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:04.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:04.002+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.004132+0000) 2024-02-19T20:16:04.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:04.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:04.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:04.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:04.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:04.024+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:04.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:03.875+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.876130+0000) 2024-02-19T20:16:04.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:03.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:04.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:03.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:04.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:03.927+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:04.876 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:04.783+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.785233+0000) 2024-02-19T20:16:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:04.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:04.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:04.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:04.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:04.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:05.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:05.006+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.007049+0000) 2024-02-19T20:16:05.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:05.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:05.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:05.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:05.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:05.042+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:05.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:04.874+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.876274+0000) 2024-02-19T20:16:05.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:04.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:05.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:04.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:05.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:04.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:05.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:05.784+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.785487+0000) 2024-02-19T20:16:05.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:05.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:05.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:05.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:05.877 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:05.597+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:05.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:05.597+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:06.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:06.008+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.008890+0000) 2024-02-19T20:16:06.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:06.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:06.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:06.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:06.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:06.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:06.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:05.875+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.876429+0000) 2024-02-19T20:16:06.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:05.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:06.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:05.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:06.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:05.912+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:06.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:06.784+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.785638+0000) 2024-02-19T20:16:06.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:06.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:06.898 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:06.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:06.898 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:06.626+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:06.898 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:06.626+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:06.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:06.875+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.876619+0000) 2024-02-19T20:16:06.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:06.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:06.898 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:06.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:07.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:06.895+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:07.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:07.008+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.009133+0000) 2024-02-19T20:16:07.199 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:07.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:07.199 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:07.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:07.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:07.079+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:07.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:07.785+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.785818+0000) 2024-02-19T20:16:07.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:07.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:07.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:07.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:07.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:07.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:07.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:07.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:07.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:07.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:07.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:07.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:07.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:07.860+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:07.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:07.876+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.876809+0000) 2024-02-19T20:16:08.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:07.989+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.991158+0000) 2024-02-19T20:16:08.196 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:07.996+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53559 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:08.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:07.993+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.994765+0000) 2024-02-19T20:16:08.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:08.007+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.009380+0000) 2024-02-19T20:16:08.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:07.989+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.990858+0000) 2024-02-19T20:16:08.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:08.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:08.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:08.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:08.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:08.032+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:08.942 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:08.784+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.785964+0000) 2024-02-19T20:16:08.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:08.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:08.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:08.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:08.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:08.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:08.943 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:08.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:08.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:08.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:08.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:08.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:08.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:08.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:08.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:08.876+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.877035+0000) 2024-02-19T20:16:09.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:09.011+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.012650+0000) 2024-02-19T20:16:09.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:09.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:09.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:09.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:09.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:09.040+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:09.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:09.785+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.786161+0000) 2024-02-19T20:16:09.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:09.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:09.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:09.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:09.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:09.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:09.943 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:09.876+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.877215+0000) 2024-02-19T20:16:09.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:09.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:09.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:09.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:09.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:09.630+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:09.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:09.630+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:10.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:10.013+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.015392+0000) 2024-02-19T20:16:10.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:10.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:10.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:10.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:10.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:10.012+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:10.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:10.785+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.786383+0000) 2024-02-19T20:16:10.878 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:10.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:10.878 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:10.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:10.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:10.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:10.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:10.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:10.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:10.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:10.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:10.605+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:10.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:10.605+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:11.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:10.876+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.877416+0000) 2024-02-19T20:16:11.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:11.016+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.017439+0000) 2024-02-19T20:16:11.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:11.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:11.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:11.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:11.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:11.019+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:11.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:11.786+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.786651+0000) 2024-02-19T20:16:11.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:11.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:11.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:11.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:11.878 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:11.591+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:11.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:11.591+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:12.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:12.017+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.017807+0000) 2024-02-19T20:16:12.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:12.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:12.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:12.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:12.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:12.034+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:12.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:11.876+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.877608+0000) 2024-02-19T20:16:12.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:11.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:12.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:11.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:12.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:11.894+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:12.869 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:12.786+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.786883+0000) 2024-02-19T20:16:12.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:12.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:12.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:12.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:12.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:12.563+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:12.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:12.563+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:13.126 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:13.005+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53632 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:13.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:13.002+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.003665+0000) 2024-02-19T20:16:13.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:13.016+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.018066+0000) 2024-02-19T20:16:13.127 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:12.998+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.999762+0000) 2024-02-19T20:16:13.127 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:13.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:13.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:13.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:13.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:13.048+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:13.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:12.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:13.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:12.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:13.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:12.867+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:13.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:12.876+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.877781+0000) 2024-02-19T20:16:13.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:12.998+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.999828+0000) 2024-02-19T20:16:13.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:13.786+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.787082+0000) 2024-02-19T20:16:13.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:13.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:13.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:13.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:13.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:13.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:13.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:13.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:13.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:13.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:13.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:13.583+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:13.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:13.583+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:14.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:14.020+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.020881+0000) 2024-02-19T20:16:14.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:13.877+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.878003+0000) 2024-02-19T20:16:14.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:14.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:14.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:14.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:14.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:14.059+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:14.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:14.785+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.787180+0000) 2024-02-19T20:16:14.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:14.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:14.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:14.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:14.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:14.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:14.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:14.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:14.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:14.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:14.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:14.610+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:14.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:14.610+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:15.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:14.877+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.878111+0000) 2024-02-19T20:16:15.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:15.022+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.023486+0000) 2024-02-19T20:16:15.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:15.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:15.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:15.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:15.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:15.078+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:15.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:15.786+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.787371+0000) 2024-02-19T20:16:15.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:15.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:15.944 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:15.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:15.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:15.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:15.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:15.877+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.878241+0000) 2024-02-19T20:16:15.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:15.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:15.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:15.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:15.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:15.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:15.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:15.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:16.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:16.024+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.025481+0000) 2024-02-19T20:16:16.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:16.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:16.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:16.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:16.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:16.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:16.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:16.787+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.787544+0000) 2024-02-19T20:16:16.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:16.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:16.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:16.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:16.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:16.828+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:16.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:16.877+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.878406+0000) 2024-02-19T20:16:16.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:16.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:16.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:16.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:16.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:16.675+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:16.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:16.676+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:17.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:17.024+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.025788+0000) 2024-02-19T20:16:17.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:17.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:17.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:17.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:17.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:17.083+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:17.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:17.787+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.787687+0000) 2024-02-19T20:16:17.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:17.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:17.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:17.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:17.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:17.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:17.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:17.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:17.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:17.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:17.945 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:17.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:17.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:17.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:17.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:17.878+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.878555+0000) 2024-02-19T20:16:18.197 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:18.013+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53705 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:18.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:18.008+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.009676+0000) 2024-02-19T20:16:18.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:18.011+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.012585+0000) 2024-02-19T20:16:18.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:18.024+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.026078+0000) 2024-02-19T20:16:18.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:18.007+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.008656+0000) 2024-02-19T20:16:18.199 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:18.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:18.199 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:18.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:18.199 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:18.112+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:18.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:18.787+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.787899+0000) 2024-02-19T20:16:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:18.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:18.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:18.643+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:18.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:18.643+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:18.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:18.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:18.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:18.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:18.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:18.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:18.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:18.878+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.878740+0000) 2024-02-19T20:16:19.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:19.028+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.028735+0000) 2024-02-19T20:16:19.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:19.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:19.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:19.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:19.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:19.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:19.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:19.787+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.788052+0000) 2024-02-19T20:16:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:19.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:19.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:19.680+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:19.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:19.681+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:19.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:19.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:19.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:19.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:19.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:19.784+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:19.946 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:19.878+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.878823+0000) 2024-02-19T20:16:20.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:20.030+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.031373+0000) 2024-02-19T20:16:20.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:20.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:20.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:20.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:20.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:20.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:20.787+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.788238+0000) 2024-02-19T20:16:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:20.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:20.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:20.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:20.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:20.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:20.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:20.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:20.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:20.878+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.878975+0000) 2024-02-19T20:16:21.358 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:21.033+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.033492+0000) 2024-02-19T20:16:21.358 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:21.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:21.358 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:21.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:21.359 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:21.109+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:21.788+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.788481+0000) 2024-02-19T20:16:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:21.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:21.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:21.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:21.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:21.721+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:21.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:21.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:21.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:21.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:21.779+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:21.981 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:21.878+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.879190+0000) 2024-02-19T20:16:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:22.033+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.033721+0000) 2024-02-19T20:16:22.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:22.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:22.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:22.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:22.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:22.069+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:22.788+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.788697+0000) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:22.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:22.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:22.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:22.719+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:22.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:22.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:22.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:22.819+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:22.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:22.878+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.879346+0000) 2024-02-19T20:16:23.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:23.023+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53778 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:23.016+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.016648+0000) 2024-02-19T20:16:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:23.019+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.019821+0000) 2024-02-19T20:16:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:23.033+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.033950+0000) 2024-02-19T20:16:23.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:23.016+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.016474+0000) 2024-02-19T20:16:23.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:23.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:23.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:23.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:23.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:23.041+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:23.788+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.788873+0000) 2024-02-19T20:16:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:23.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:23.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:23.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:23.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:23.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:23.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:23.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:23.775+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:23.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:23.879+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.879512+0000) 2024-02-19T20:16:24.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:24.036+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.036623+0000) 2024-02-19T20:16:24.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:24.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:24.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:24.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:24.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:24.017+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:25.040 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:24.788+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.789021+0000) 2024-02-19T20:16:25.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:24.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:25.040 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:24.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:25.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:24.765+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:25.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:24.879+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.879658+0000) 2024-02-19T20:16:25.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:24.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:25.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:24.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:25.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:24.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:25.041 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:24.736+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:25.038+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.039288+0000) 2024-02-19T20:16:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:25.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:25.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:25.053+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:25.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:25.788+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.789184+0000) 2024-02-19T20:16:25.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:25.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:25.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:25.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:25.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:25.697+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:25.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:25.697+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:25.973 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:25.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:25.973 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:25.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:25.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:25.740+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:25.974 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:25.879+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.879848+0000) 2024-02-19T20:16:26.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:26.041+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.041598+0000) 2024-02-19T20:16:26.226 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:26.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:26.227 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.016+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:27.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.788+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.789361+0000) 2024-02-19T20:16:27.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:27.043 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:27.043 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:26.975+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:27.043 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:26.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:27.043 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:26.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:27.043 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:26.729+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:27.043 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:26.879+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.879996+0000) 2024-02-19T20:16:27.044 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:26.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:27.044 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:26.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:27.044 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:26.736+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:27.044 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:26.736+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:27.044 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:27.041+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.041802+0000) 2024-02-19T20:16:27.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:27.789+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.789553+0000) 2024-02-19T20:16:27.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:27.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:27.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:27.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:27.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:27.879+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.880194+0000) 2024-02-19T20:16:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:27.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:27.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:27.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:27.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:27.980 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:27.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:28.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:28.031+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53851 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:28.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:28.025+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.026247+0000) 2024-02-19T20:16:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:28.028+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.029068+0000) 2024-02-19T20:16:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:28.041+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.041995+0000) 2024-02-19T20:16:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:28.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:28.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:28.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:28.007+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:28.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:28.025+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.026027+0000) 2024-02-19T20:16:28.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:28.789+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.789672+0000) 2024-02-19T20:16:28.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:28.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:28.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:28.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:28.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:28.774+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:28.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:28.880+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.880387+0000) 2024-02-19T20:16:28.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:28.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:28.947 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:28.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:28.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:28.659+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:28.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:28.660+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:29.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:29.044+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.044744+0000) 2024-02-19T20:16:29.199 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:29.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:29.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:29.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:29.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:29.022+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:29.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:29.789+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.789832+0000) 2024-02-19T20:16:29.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:29.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:29.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:29.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:29.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:29.801+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:29.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:29.880+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.880535+0000) 2024-02-19T20:16:29.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:29.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:29.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:29.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:29.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:29.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:29.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:29.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:30.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:30.047+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.047331+0000) 2024-02-19T20:16:30.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:30.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:30.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:30.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:30.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:30.064+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:30.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:30.789+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.789995+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:30.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:30.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:30.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:30.880+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.880673+0000) 2024-02-19T20:16:30.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:30.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:30.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:30.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:30.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:30.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:30.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:30.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:31.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:31.049+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.049537+0000) 2024-02-19T20:16:31.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:31.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:31.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:31.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:31.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:31.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:31.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:31.789+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.790166+0000) 2024-02-19T20:16:31.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:31.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:31.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:31.868+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:31.880+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.880856+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:31.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:31.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:31.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:31.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:31.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:32.049+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.049848+0000) 2024-02-19T20:16:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:32.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:32.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:32.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:32.068+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:32.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:32.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.790361+0000) 2024-02-19T20:16:32.947 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:32.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:32.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:32.834+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:32.880+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.881054+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:32.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:32.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:32.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:32.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:32.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:33.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:33.033+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.034067+0000) 2024-02-19T20:16:33.201 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:33.040+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53924 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:33.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:33.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:33.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:33.028+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:33.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:33.029+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:33.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:33.033+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.033993+0000) 2024-02-19T20:16:33.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:33.037+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.037999+0000) 2024-02-19T20:16:33.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:33.049+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.050077+0000) 2024-02-19T20:16:33.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:33.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.790537+0000) 2024-02-19T20:16:33.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:33.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:33.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:33.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:33.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:33.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:33.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:33.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:33.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:33.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:33.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:33.814+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:33.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:33.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.881238+0000) 2024-02-19T20:16:34.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:34.052+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.053107+0000) 2024-02-19T20:16:34.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:34.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:34.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:34.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:34.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:34.038+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:34.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:34.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.790659+0000) 2024-02-19T20:16:34.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:34.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:34.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:34.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:34.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:34.859+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:34.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:34.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.881414+0000) 2024-02-19T20:16:34.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:34.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:34.948 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:34.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:34.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:34.657+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:34.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:34.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:35.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:35.055+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.055917+0000) 2024-02-19T20:16:35.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:35.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:35.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:35.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:35.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:35.071+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:35.947 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:35.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.790812+0000) 2024-02-19T20:16:35.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:35.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:35.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:35.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:35.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:35.851+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:35.948 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:35.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.881561+0000) 2024-02-19T20:16:35.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:35.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:35.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:35.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:35.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:35.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:35.949 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:35.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:36.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:36.057+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.057910+0000) 2024-02-19T20:16:36.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:36.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:36.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:36.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:36.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:36.031+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:36.791 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:36.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:36.792 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:36.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:36.792 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:36.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:36.792 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:36.665+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:37.058 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:36.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:37.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:36.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:37.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:36.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:37.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:36.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.881770+0000) 2024-02-19T20:16:37.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:36.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.791009+0000) 2024-02-19T20:16:37.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:37.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:37.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:37.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:37.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:37.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:37.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:37.058+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.058140+0000) 2024-02-19T20:16:37.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:37.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.791220+0000) 2024-02-19T20:16:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:37.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:37.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:37.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:37.820+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:37.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:37.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.881929+0000) 2024-02-19T20:16:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:37.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:37.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:37.703+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:37.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:37.703+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:38.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:38.048+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 53997 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:38.041+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.042325+0000) 2024-02-19T20:16:38.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:38.045+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.045544+0000) 2024-02-19T20:16:38.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:38.057+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.058342+0000) 2024-02-19T20:16:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:38.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:38.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:38.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:38.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:38.030+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:38.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:38.042+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.042746+0000) 2024-02-19T20:16:38.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:38.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.791432+0000) 2024-02-19T20:16:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:38.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:38.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:38.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:38.714+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:38.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:38.714+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:38.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:38.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:38.805+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:38.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:38.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.882072+0000) 2024-02-19T20:16:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:39.060+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.061124+0000) 2024-02-19T20:16:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:39.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:39.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:39.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:39.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:39.060+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:40.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:39.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.791609+0000) 2024-02-19T20:16:40.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:39.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:40.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:39.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:40.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:39.833+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:40.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:39.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.882241+0000) 2024-02-19T20:16:40.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:39.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:40.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:39.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:40.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:39.757+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:40.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:39.757+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:40.063+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.063899+0000) 2024-02-19T20:16:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:40.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:40.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:40.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:40.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:40.070+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:41.066 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:40.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.791799+0000) 2024-02-19T20:16:41.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:40.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:41.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:40.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:41.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:40.818+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:41.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:40.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.882431+0000) 2024-02-19T20:16:41.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:40.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:41.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:40.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:41.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:40.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:41.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:40.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:41.065+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.066227+0000) 2024-02-19T20:16:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:41.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:41.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:41.118+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:41.792 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:41.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:41.792 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:41.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:41.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:41.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:41.793 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:41.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:42.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:41.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.792004+0000) 2024-02-19T20:16:42.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:41.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:42.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:41.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:42.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:41.796+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:42.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:41.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.882665+0000) 2024-02-19T20:16:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:42.065+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.066556+0000) 2024-02-19T20:16:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:42.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:42.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:42.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:42.949 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:42.790+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.792214+0000) 2024-02-19T20:16:42.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:42.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:42.949 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:42.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:42.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:42.755+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:42.950 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:42.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.882842+0000) 2024-02-19T20:16:42.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:42.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:42.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:42.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:42.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:42.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:42.950 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:42.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:43.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:43.048+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.050272+0000) 2024-02-19T20:16:43.202 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:43.055+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54070 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:43.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:43.053+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.054506+0000) 2024-02-19T20:16:43.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:43.065+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.066852+0000) 2024-02-19T20:16:43.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:43.048+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.050124+0000) 2024-02-19T20:16:43.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:43.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:43.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:43.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:43.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:43.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:43.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:43.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.792410+0000) 2024-02-19T20:16:43.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:43.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:43.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:43.722+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:43.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:43.722+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:43.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:43.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:43.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:43.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:43.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:43.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.883042+0000) 2024-02-19T20:16:44.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:44.068+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.069478+0000) 2024-02-19T20:16:44.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:44.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:44.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:44.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:44.404 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:44.102+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:45.073 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:44.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.792552+0000) 2024-02-19T20:16:45.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:44.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:45.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:44.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:45.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:44.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:45.074 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:44.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:45.074 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:44.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:45.074 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:44.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:45.075 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:44.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:45.075 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:44.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.883183+0000) 2024-02-19T20:16:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:45.071+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.072578+0000) 2024-02-19T20:16:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:45.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:45.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:45.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:45.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:46.075 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:45.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.792655+0000) 2024-02-19T20:16:46.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:45.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:46.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:45.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:46.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:45.835+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:46.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:45.881+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.883358+0000) 2024-02-19T20:16:46.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:45.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:46.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:45.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:46.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:45.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:46.077 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:45.791+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:46.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:46.074+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.075091+0000) 2024-02-19T20:16:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:46.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:46.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:46.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:46.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:46.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:46.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:46.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.792801+0000) 2024-02-19T20:16:46.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:46.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:46.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:46.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:46.884 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:46.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:46.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:46.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:46.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:46.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:46.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:46.762+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:46.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:46.762+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:47.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:47.073+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.075369+0000) 2024-02-19T20:16:47.233 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:46.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.883526+0000) 2024-02-19T20:16:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:47.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:47.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:47.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:47.199+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:48.059 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:47.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.792962+0000) 2024-02-19T20:16:48.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:47.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:48.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:47.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:48.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:47.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:48.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:47.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.883722+0000) 2024-02-19T20:16:48.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:47.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:48.060 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:47.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:48.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:47.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:48.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:47.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:48.057+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.058716+0000) 2024-02-19T20:16:48.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:48.063+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54144 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:48.060+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.062052+0000) 2024-02-19T20:16:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:48.074+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.075644+0000) 2024-02-19T20:16:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:48.057+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.058817+0000) 2024-02-19T20:16:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:48.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:48.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:48.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:48.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:48.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:49.079 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:48.791+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.793187+0000) 2024-02-19T20:16:49.080 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:48.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:49.080 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:48.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:49.080 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:48.770+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:49.081 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:48.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.883861+0000) 2024-02-19T20:16:49.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:48.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:49.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:48.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:49.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:48.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:49.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:48.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:49.081 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:49.077+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.079025+0000) 2024-02-19T20:16:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:49.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:49.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:49.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:49.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:49.792+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.793338+0000) 2024-02-19T20:16:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:49.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:49.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:49.749+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:49.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.884028+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:49.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:49.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:49.719+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:49.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:49.719+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:50.080+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.081894+0000) 2024-02-19T20:16:50.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:50.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:50.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:50.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:50.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:50.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:50.792+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.793516+0000) 2024-02-19T20:16:50.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:50.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:50.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:50.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:50.951 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:50.794+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:50.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:50.883+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.884202+0000) 2024-02-19T20:16:50.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:50.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:50.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:50.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:50.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:50.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:50.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:50.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:51.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:51.082+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.083855+0000) 2024-02-19T20:16:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:51.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:51.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:51.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:51.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:51.884 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:51.792+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.793712+0000) 2024-02-19T20:16:51.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:51.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:51.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:51.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:51.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:51.809+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:51.885 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:51.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:51.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:51.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:51.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:51.695+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:51.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:51.695+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:52.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:52.083+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.084128+0000) 2024-02-19T20:16:52.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:51.882+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.884367+0000) 2024-02-19T20:16:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:52.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:52.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:52.198+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:52.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:52.793+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.793909+0000) 2024-02-19T20:16:52.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:52.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:52.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:52.711+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:52.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:52.711+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:52.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:52.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:52.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:52.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:52.766+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:52.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:52.883+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.884577+0000) 2024-02-19T20:16:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:53.066+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.067636+0000) 2024-02-19T20:16:53.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:53.072+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54217 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:53.068+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.070023+0000) 2024-02-19T20:16:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:53.082+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.084359+0000) 2024-02-19T20:16:53.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:53.065+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.066727+0000) 2024-02-19T20:16:53.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:53.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:53.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:53.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:53.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:53.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:53.792+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.794034+0000) 2024-02-19T20:16:53.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:53.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:53.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:53.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:53.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:53.666+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:53.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:53.666+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:53.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:53.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:53.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:53.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:53.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:53.790+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:53.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:53.884+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.884767+0000) 2024-02-19T20:16:54.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:54.086+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.087194+0000) 2024-02-19T20:16:54.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:54.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:54.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:54.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:54.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:54.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:54.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:54.793+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.794199+0000) 2024-02-19T20:16:54.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:54.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:54.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:54.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:54.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:54.751+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:54.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:54.883+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.884886+0000) 2024-02-19T20:16:54.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:54.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:54.952 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:54.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:54.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:54.685+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:54.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:54.685+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:55.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:55.089+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.090004+0000) 2024-02-19T20:16:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:55.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:55.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:55.223+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:55.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:55.793+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.794331+0000) 2024-02-19T20:16:55.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:55.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:55.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:55.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:55.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:55.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:55.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:55.884+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.885072+0000) 2024-02-19T20:16:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:55.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:55.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:55.637+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:55.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:55.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:56.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:56.090+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.092171+0000) 2024-02-19T20:16:56.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:56.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:56.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:56.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:56.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:56.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:56.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:56.793+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.794530+0000) 2024-02-19T20:16:56.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:56.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:56.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:56.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:56.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:56.746+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:56.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:56.884+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.885244+0000) 2024-02-19T20:16:56.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:56.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:56.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:56.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:56.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:56.678+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:56.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:56.678+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:57.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:57.091+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.092490+0000) 2024-02-19T20:16:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:57.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:57.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:57.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:57.218+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:57.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:57.793+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.794730+0000) 2024-02-19T20:16:57.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:57.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:57.791+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:57.884+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.885426+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:57.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:57.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:57.669+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:57.669+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:58.203 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:16:58.080+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54290 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:16:58.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:58.074+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.075692+0000) 2024-02-19T20:16:58.204 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:58.074+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.075776+0000) 2024-02-19T20:16:58.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:58.077+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.078959+0000) 2024-02-19T20:16:58.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:58.091+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.092707+0000) 2024-02-19T20:16:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:58.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:58.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:58.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:58.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:58.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:58.794+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.794904+0000) 2024-02-19T20:16:58.952 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:58.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:58.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:58.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:58.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:58.829+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:58.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:58.884+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.885612+0000) 2024-02-19T20:16:58.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:58.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:58.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:58.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:58.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:58.645+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:58.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:58.645+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:59.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:59.093+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.095193+0000) 2024-02-19T20:16:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:59.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:59.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:59.216+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:16:59.952 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:16:59.794+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.795046+0000) 2024-02-19T20:16:59.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:59.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:59.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:59.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:59.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:59.641+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:16:59.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:16:59.641+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:59.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:59.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:59.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:59.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:59.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:59.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:16:59.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:16:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:16:59.885+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.885770+0000) 2024-02-19T20:17:00.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:00.098+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.098917+0000) 2024-02-19T20:17:00.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:00.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:00.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:00.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:00.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:00.196+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:00.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:00.793+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.795258+0000) 2024-02-19T20:17:00.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:00.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:00.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:00.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:00.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:00.793+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:00.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:00.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:00.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:00.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:00.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:00.633+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:00.888 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:00.633+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:01.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:01.099+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.100914+0000) 2024-02-19T20:17:01.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:00.885+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.885975+0000) 2024-02-19T20:17:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:01.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:01.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:01.185+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:01.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:01.794+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.795471+0000) 2024-02-19T20:17:01.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:01.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:01.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:01.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:01.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:01.609+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:01.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:01.609+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:01.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:01.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:01.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:01.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:01.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:01.788+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:02.204 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:02.099+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.101153+0000) 2024-02-19T20:17:02.205 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:01.885+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.886221+0000) 2024-02-19T20:17:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:02.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:02.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:02.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:02.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:02.795+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.795653+0000) 2024-02-19T20:17:02.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:02.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:02.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:02.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:02.953 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:02.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:02.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:02.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:02.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:02.823+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:02.885+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.886416+0000) 2024-02-19T20:17:03.205 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:03.089+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54363 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:03.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:03.082+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.083750+0000) 2024-02-19T20:17:03.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:03.081+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.083293+0000) 2024-02-19T20:17:03.206 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:03.085+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.086948+0000) 2024-02-19T20:17:03.206 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:03.099+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.101353+0000) 2024-02-19T20:17:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:03.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:03.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:03.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:03.794+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.795758+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:03.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:03.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:03.854+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:03.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:03.886+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.886604+0000) 2024-02-19T20:17:03.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:03.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:03.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:03.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:03.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:03.664+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:03.954 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:03.664+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:04.205 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:04.103+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.103922+0000) 2024-02-19T20:17:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:04.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:04.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:04.274+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:04.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:04.794+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.795843+0000) 2024-02-19T20:17:04.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:04.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:04.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:04.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:04.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:04.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:04.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:04.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:04.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:04.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:04.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:04.815+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:04.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:04.886+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.886723+0000) 2024-02-19T20:17:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:05.107+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.107554+0000) 2024-02-19T20:17:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:05.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:05.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:05.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:05.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:05.236+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:06.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:05.795+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.796041+0000) 2024-02-19T20:17:06.110 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:05.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:06.110 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:05.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:06.111 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:05.806+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:06.111 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:05.886+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.886900+0000) 2024-02-19T20:17:06.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:05.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:06.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:05.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:06.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:05.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:06.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:05.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:06.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:06.109+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.109545+0000) 2024-02-19T20:17:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:06.219+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:06.219+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:06.219+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:07.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:06.795+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.796193+0000) 2024-02-19T20:17:07.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:06.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:07.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:06.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:07.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:06.787+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:07.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:06.787+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:07.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:06.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:07.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:06.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:07.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:06.789+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:07.112 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:06.885+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.887024+0000) 2024-02-19T20:17:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:07.109+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.109901+0000) 2024-02-19T20:17:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:07.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:07.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:07.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:07.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:08.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:07.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:08.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:07.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:08.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:07.821+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:08.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:07.886+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.887136+0000) 2024-02-19T20:17:08.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:07.795+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.796362+0000) 2024-02-19T20:17:08.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:07.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:08.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:07.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:08.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:07.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:08.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:07.770+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:08.092+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.092843+0000) 2024-02-19T20:17:08.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:08.099+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54436 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:08.094+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.095148+0000) 2024-02-19T20:17:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:08.109+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.110142+0000) 2024-02-19T20:17:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:08.092+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.092526+0000) 2024-02-19T20:17:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:08.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:08.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:08.277+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:08.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:08.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.796567+0000) 2024-02-19T20:17:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:08.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:08.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:08.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:08.804+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:08.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:08.886+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.887298+0000) 2024-02-19T20:17:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:08.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:08.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:08.724+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:08.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:08.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:09.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:09.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:09.273+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:09.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:09.112+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.113269+0000) 2024-02-19T20:17:10.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:09.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.796694+0000) 2024-02-19T20:17:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:09.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:09.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:09.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:10.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:09.756+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:10.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:09.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:10.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:09.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:10.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:09.845+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:10.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:09.887+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.887472+0000) 2024-02-19T20:17:10.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:10.115+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.115920+0000) 2024-02-19T20:17:10.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:10.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:10.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:10.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:10.238+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:11.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:10.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.796796+0000) 2024-02-19T20:17:11.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:10.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:11.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:10.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:11.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:10.755+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:11.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:10.755+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:11.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:10.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:11.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:10.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:11.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:10.885+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:11.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:10.887+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.887603+0000) 2024-02-19T20:17:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:11.118+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.118587+0000) 2024-02-19T20:17:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:11.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:11.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:11.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:11.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:11.284+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:12.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:11.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.796934+0000) 2024-02-19T20:17:12.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:11.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:12.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:11.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:12.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:11.875+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:12.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:11.887+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.887733+0000) 2024-02-19T20:17:12.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:11.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:12.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:11.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:12.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:11.752+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:12.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:11.752+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:12.118+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.118826+0000) 2024-02-19T20:17:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:12.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:12.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:12.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:12.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:12.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:13.101 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:12.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.797107+0000) 2024-02-19T20:17:13.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:12.887+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.887866+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:12.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:12.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:12.890+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:12.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:12.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:12.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:13.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:12.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:13.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:13.106+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54509 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:13.100+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.100859+0000) 2024-02-19T20:17:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:13.100+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.100984+0000) 2024-02-19T20:17:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:13.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:13.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:13.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:13.234+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:13.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:13.103+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.103932+0000) 2024-02-19T20:17:13.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:13.118+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.119048+0000) 2024-02-19T20:17:14.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:13.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.797271+0000) 2024-02-19T20:17:14.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:13.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:14.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:13.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:14.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:13.864+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:14.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:13.887+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.887987+0000) 2024-02-19T20:17:14.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:13.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:14.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:13.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:14.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:13.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:14.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:13.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:14.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:14.121+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.121633+0000) 2024-02-19T20:17:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:14.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:14.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:14.195+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:15.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:14.796+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.797432+0000) 2024-02-19T20:17:15.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:14.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:15.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:14.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:15.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:14.846+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:15.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:14.887+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.888150+0000) 2024-02-19T20:17:15.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:14.764+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:15.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:14.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:15.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:14.765+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:15.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:14.765+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:15.123+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.124148+0000) 2024-02-19T20:17:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:15.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:15.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:15.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:15.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:15.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:16.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:15.797+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.797547+0000) 2024-02-19T20:17:16.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:15.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:16.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:15.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:16.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:15.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:16.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:15.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:16.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:15.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.888343+0000) 2024-02-19T20:17:16.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:15.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:16.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:15.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:16.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:15.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:16.125+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.126133+0000) 2024-02-19T20:17:16.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:16.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:16.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:16.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:17.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:16.797+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.797675+0000) 2024-02-19T20:17:17.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:16.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:17.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:16.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:17.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:16.746+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:17.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:16.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:17.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:16.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.888524+0000) 2024-02-19T20:17:17.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:16.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:17.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:16.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:17.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:16.918+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:17.125+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.126388+0000) 2024-02-19T20:17:17.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:17.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:17.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:17.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:17.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:17.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:18.109 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:17.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.888653+0000) 2024-02-19T20:17:18.109 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:17.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:18.109 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:17.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:18.110 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:17.935+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:18.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:17.797+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.797859+0000) 2024-02-19T20:17:18.110 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:17.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:18.110 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:17.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:18.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:17.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:18.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:17.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:18.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:18.115+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54582 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:18.108+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.108641+0000) 2024-02-19T20:17:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:18.112+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.112758+0000) 2024-02-19T20:17:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:18.126+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.126663+0000) 2024-02-19T20:17:18.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:18.108+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.108972+0000) 2024-02-19T20:17:18.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:18.168+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:18.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:18.168+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:18.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:18.168+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:19.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:18.797+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.798071+0000) 2024-02-19T20:17:19.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:18.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.888825+0000) 2024-02-19T20:17:19.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:18.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:19.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:18.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:19.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:18.973+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:19.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:18.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:19.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:18.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:19.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:18.758+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:19.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:18.758+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:19.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:19.128+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.129079+0000) 2024-02-19T20:17:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:19.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:19.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:19.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:20.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:19.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.798245+0000) 2024-02-19T20:17:20.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:19.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:20.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:19.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:20.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:19.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:20.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:19.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:20.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:19.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.888984+0000) 2024-02-19T20:17:20.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:19.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:20.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:19.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:20.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:19.951+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:20.131+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.131776+0000) 2024-02-19T20:17:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:20.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:20.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:20.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:20.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:20.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:20.797+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.798402+0000) 2024-02-19T20:17:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:20.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:20.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:20.713+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:20.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:20.713+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:20.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.889152+0000) 2024-02-19T20:17:20.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:20.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:20.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:20.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:20.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:20.907+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:21.133+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.133823+0000) 2024-02-19T20:17:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:21.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:21.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:21.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:21.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:21.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:21.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:21.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.798581+0000) 2024-02-19T20:17:21.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:21.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:21.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:21.720+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:21.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:21.720+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:21.888+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.889280+0000) 2024-02-19T20:17:21.979 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:21.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:21.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:21.980 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:21.956+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:22.133+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.134194+0000) 2024-02-19T20:17:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:22.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:22.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:22.090+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:22.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:22.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.798748+0000) 2024-02-19T20:17:22.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:22.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.889486+0000) 2024-02-19T20:17:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:22.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:22.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:22.706+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:22.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:22.706+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:23.248 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:23.123+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54655 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:23.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:23.120+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.121142+0000) 2024-02-19T20:17:23.248 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:23.133+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.134428+0000) 2024-02-19T20:17:23.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:22.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:23.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:22.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:23.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:22.985+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:23.249 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:23.117+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.117438+0000) 2024-02-19T20:17:23.250 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:23.117+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.118402+0000) 2024-02-19T20:17:23.250 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:23.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:23.250 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:23.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:23.250 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:23.131+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:23.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:23.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.798908+0000) 2024-02-19T20:17:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:23.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.889688+0000) 2024-02-19T20:17:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:23.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:23.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:23.726+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:23.979 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:23.726+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:24.268 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:24.136+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.136913+0000) 2024-02-19T20:17:24.269 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:24.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:24.269 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:24.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:24.269 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:24.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:24.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:24.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:24.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:24.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:24.269 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:24.179+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:25.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:24.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.799062+0000) 2024-02-19T20:17:25.058 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:24.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.889849+0000) 2024-02-19T20:17:25.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:24.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:25.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:24.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:25.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:24.776+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:25.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:24.776+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:25.139+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.139624+0000) 2024-02-19T20:17:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:25.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:25.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:25.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:25.174+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:25.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:25.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:25.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:25.056+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:26.072 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:25.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.799258+0000) 2024-02-19T20:17:26.072 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:25.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.890048+0000) 2024-02-19T20:17:26.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:25.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:26.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:25.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:26.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:25.784+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:26.073 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:25.784+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:26.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:26.141+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.141661+0000) 2024-02-19T20:17:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:26.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:26.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:26.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:26.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:26.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:26.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:26.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:26.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:26.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:26.071+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:27.106 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:26.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.799438+0000) 2024-02-19T20:17:27.106 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:26.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.890252+0000) 2024-02-19T20:17:27.107 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:27.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:27.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:26.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:27.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:26.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:27.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:26.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:27.107 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:26.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:27.141+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.141978+0000) 2024-02-19T20:17:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:27.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:27.105+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:27.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:27.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:27.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:27.156+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:28.117 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:27.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.799610+0000) 2024-02-19T20:17:28.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:27.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.890402+0000) 2024-02-19T20:17:28.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:27.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:28.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:27.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:28.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:27.754+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:28.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:27.754+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:28.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:28.131+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54722 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:28.128+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.129267+0000) 2024-02-19T20:17:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:28.141+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.142237+0000) 2024-02-19T20:17:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:28.125+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.125945+0000) 2024-02-19T20:17:28.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:28.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:28.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:28.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:28.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:28.158+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:28.126+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.126746+0000) 2024-02-19T20:17:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:28.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:28.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:28.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:28.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:29.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:28.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.799744+0000) 2024-02-19T20:17:29.110 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:28.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.890625+0000) 2024-02-19T20:17:29.110 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:29.109+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:29.110 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:28.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:29.110 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:28.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:29.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:28.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:29.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:28.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:29.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:29.144+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.145260+0000) 2024-02-19T20:17:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:29.109+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:29.109+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:29.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:29.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:29.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:29.173+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:30.117 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:29.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.799900+0000) 2024-02-19T20:17:30.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:29.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.890778+0000) 2024-02-19T20:17:30.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:29.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:30.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:29.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:30.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:29.786+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:30.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:29.786+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:30.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:30.146+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.147986+0000) 2024-02-19T20:17:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:30.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:30.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:30.140+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:30.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:30.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:30.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:30.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:30.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:30.137+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:31.117 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:30.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.800008+0000) 2024-02-19T20:17:31.117 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:30.889+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.890947+0000) 2024-02-19T20:17:31.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:30.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:31.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:30.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:31.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:30.821+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:31.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:30.821+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:31.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:31.148+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.150365+0000) 2024-02-19T20:17:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:31.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:31.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:31.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:31.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:31.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:31.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:31.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:31.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:31.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:32.117 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:31.798+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.800179+0000) 2024-02-19T20:17:32.118 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:31.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.891120+0000) 2024-02-19T20:17:32.118 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:31.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:32.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:31.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:32.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:31.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:32.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:31.869+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:32.149+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.150602+0000) 2024-02-19T20:17:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:32.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:32.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:32.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:32.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:32.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:32.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:32.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:32.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:32.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:32.162+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:33.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:32.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.800368+0000) 2024-02-19T20:17:33.121 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:32.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.891303+0000) 2024-02-19T20:17:33.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:32.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:33.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:32.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:33.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:32.918+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:33.121 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:32.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:33.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:33.138+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54789 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:33.135+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.137152+0000) 2024-02-19T20:17:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:33.149+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.150925+0000) 2024-02-19T20:17:33.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:33.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:33.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:33.116+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:33.132+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.133560+0000) 2024-02-19T20:17:33.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:33.132+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.134012+0000) 2024-02-19T20:17:33.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:33.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:33.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:33.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:33.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:33.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:34.078 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:33.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.891509+0000) 2024-02-19T20:17:34.079 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:33.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.800580+0000) 2024-02-19T20:17:34.079 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:34.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:34.079 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:34.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:34.079 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:34.076+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:34.079 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:33.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:34.079 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:33.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:34.080 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:33.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:34.080 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:33.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:34.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:34.153+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.154475+0000) 2024-02-19T20:17:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:34.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:34.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:34.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:35.110 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:34.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.891624+0000) 2024-02-19T20:17:35.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:34.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.800689+0000) 2024-02-19T20:17:35.110 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:35.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:35.110 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:34.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:35.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:34.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:35.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:34.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:35.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:34.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:35.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:35.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:35.108+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:35.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:35.156+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.158337+0000) 2024-02-19T20:17:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:35.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:35.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:35.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:35.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:36.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:35.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.800816+0000) 2024-02-19T20:17:36.118 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:35.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.891731+0000) 2024-02-19T20:17:36.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:35.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:36.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:35.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:36.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:35.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:36.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:35.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:36.159+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.160504+0000) 2024-02-19T20:17:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:36.197+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:36.197+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:36.197+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:36.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:36.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:36.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:36.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:36.800+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.800965+0000) 2024-02-19T20:17:37.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:36.890+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.891898+0000) 2024-02-19T20:17:37.161 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:37.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:37.162 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:37.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:37.162 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:37.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:37.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:36.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:37.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:36.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:37.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:36.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:37.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:36.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:37.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:37.159+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.160816+0000) 2024-02-19T20:17:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:37.177+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:37.177+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:37.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:37.177+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:38.058 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:37.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.892131+0000) 2024-02-19T20:17:38.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:37.800+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.801137+0000) 2024-02-19T20:17:38.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:38.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:38.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:37.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:38.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:37.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:38.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:37.893+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:38.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:37.893+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:38.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:38.145+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54858 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:38.143+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.144471+0000) 2024-02-19T20:17:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:38.159+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.161095+0000) 2024-02-19T20:17:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:38.139+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.141371+0000) 2024-02-19T20:17:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:38.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:38.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:38.168+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:38.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:38.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:38.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:38.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:38.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:38.140+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.141707+0000) 2024-02-19T20:17:39.118 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:38.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.892264+0000) 2024-02-19T20:17:39.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:38.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:39.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:38.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:39.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:38.862+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:39.119 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:38.862+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:39.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:38.799+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.801308+0000) 2024-02-19T20:17:39.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:39.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:39.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:39.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:39.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:39.037+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:39.162+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.163403+0000) 2024-02-19T20:17:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:39.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:39.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:39.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:39.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:40.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:39.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.892423+0000) 2024-02-19T20:17:40.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:39.800+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.801500+0000) 2024-02-19T20:17:40.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:40.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:40.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.000+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:40.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:39.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:40.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:39.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:40.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:39.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:40.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:39.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:40.164+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.166139+0000) 2024-02-19T20:17:40.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:40.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:40.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:40.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:41.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:40.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.892640+0000) 2024-02-19T20:17:41.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.800+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.801708+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:40.973+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:40.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:40.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:40.896+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:41.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:40.896+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:41.166+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.168301+0000) 2024-02-19T20:17:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:41.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:41.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:41.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:42.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:41.892+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.892844+0000) 2024-02-19T20:17:42.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:41.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:42.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:41.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:42.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:41.880+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:42.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:41.880+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:42.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:41.801+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.801921+0000) 2024-02-19T20:17:42.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:41.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:42.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:41.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:42.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:41.966+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:42.167+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.168648+0000) 2024-02-19T20:17:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:42.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:42.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:42.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:42.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:43.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:42.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.893044+0000) 2024-02-19T20:17:43.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:43.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:43.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:42.801+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.802060+0000) 2024-02-19T20:17:43.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:42.934+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:43.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:42.934+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:43.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:42.934+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:43.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:42.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:43.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:42.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:43.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:42.882+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:43.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:42.882+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:43.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:43.147+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.148752+0000) 2024-02-19T20:17:43.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:43.154+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54922 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:43.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:43.151+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.152633+0000) 2024-02-19T20:17:43.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:43.167+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.168938+0000) 2024-02-19T20:17:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:43.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:43.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:43.148+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.149530+0000) 2024-02-19T20:17:44.113 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:43.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.893196+0000) 2024-02-19T20:17:44.113 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:44.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:44.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:43.800+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.802242+0000) 2024-02-19T20:17:44.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:43.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:44.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:43.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:44.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:43.964+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:44.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:43.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:44.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:43.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:44.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:43.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:44.114 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:43.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:44.169+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.171197+0000) 2024-02-19T20:17:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:44.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:44.111+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:45.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:44.801+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.802407+0000) 2024-02-19T20:17:45.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:44.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:45.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:44.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:45.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:44.915+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:45.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:44.891+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.893316+0000) 2024-02-19T20:17:45.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:45.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:45.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:45.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:45.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:44.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:45.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:44.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:45.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:44.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:45.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:44.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:45.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:45.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:45.173+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.173862+0000) 2024-02-19T20:17:46.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:45.892+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.893491+0000) 2024-02-19T20:17:46.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:46.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:46.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:45.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:46.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:45.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:46.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:45.841+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:46.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:45.841+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:46.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:45.801+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.802626+0000) 2024-02-19T20:17:46.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:45.916+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:46.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:45.916+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:46.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:45.916+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:46.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:46.175+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.175921+0000) 2024-02-19T20:17:46.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:46.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:46.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:46.053+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:47.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:46.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:47.070 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:46.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:47.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:46.830+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:47.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:46.830+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:47.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:46.892+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.893682+0000) 2024-02-19T20:17:47.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:47.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:47.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:47.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:47.072 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:47.039+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:47.072 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:46.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.802799+0000) 2024-02-19T20:17:47.072 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:46.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:47.072 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:46.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:47.072 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:46.952+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:47.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:47.175+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.176222+0000) 2024-02-19T20:17:48.123 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:47.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:48.123 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:47.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:48.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:47.877+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:48.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:47.877+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:48.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:47.892+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.893789+0000) 2024-02-19T20:17:48.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:48.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:48.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:48.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:48.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:48.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:48.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:47.801+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.802940+0000) 2024-02-19T20:17:48.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:47.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:48.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:47.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:48.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:47.961+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:48.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:48.157+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.158551+0000) 2024-02-19T20:17:48.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:48.157+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.158550+0000) 2024-02-19T20:17:48.479 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:48.162+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 54990 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:48.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:48.159+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.160683+0000) 2024-02-19T20:17:48.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:48.175+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.176530+0000) 2024-02-19T20:17:49.123 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:48.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:49.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:48.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:49.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:48.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:49.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:48.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:49.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:48.893+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.893989+0000) 2024-02-19T20:17:49.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:49.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:49.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:49.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:49.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:49.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:49.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:48.801+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.803082+0000) 2024-02-19T20:17:49.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:49.126 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:49.126 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.009+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:49.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:49.178+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.179032+0000) 2024-02-19T20:17:50.124 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.803238+0000) 2024-02-19T20:17:50.124 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:50.124 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:50.124 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:49.994+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:50.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:49.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:50.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:49.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:50.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:49.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:50.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:49.900+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:50.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:49.892+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.894135+0000) 2024-02-19T20:17:50.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:50.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:50.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:50.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:50.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:50.032+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:50.181+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.181803+0000) 2024-02-19T20:17:51.123 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:50.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:51.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:50.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:51.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:50.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:51.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:50.905+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:51.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:50.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.803467+0000) 2024-02-19T20:17:51.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:50.999+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:51.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:50.999+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:51.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:50.999+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:51.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:50.893+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.894354+0000) 2024-02-19T20:17:51.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:51.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:51.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:51.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:51.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:51.044+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:51.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:51.183+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.183970+0000) 2024-02-19T20:17:52.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:51.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.803610+0000) 2024-02-19T20:17:52.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:52.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:52.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:52.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:52.094 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:52.049+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:52.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:51.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:52.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:51.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:52.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:51.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:52.095 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:51.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:52.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:51.893+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.894541+0000) 2024-02-19T20:17:52.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:52.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:52.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:52.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:52.095 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:52.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:52.183+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.184226+0000) 2024-02-19T20:17:53.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:52.894+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.894712+0000) 2024-02-19T20:17:53.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:52.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:53.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:52.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:53.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:52.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:53.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:52.917+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:53.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:52.803+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.803842+0000) 2024-02-19T20:17:53.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:53.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:53.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:53.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:53.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:53.062+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:53.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:53.164+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.165967+0000) 2024-02-19T20:17:53.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:53.171+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55060 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:53.167+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.168896+0000) 2024-02-19T20:17:53.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:53.182+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.184423+0000) 2024-02-19T20:17:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:53.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:53.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:53.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:53.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:53.164+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.166039+0000) 2024-02-19T20:17:54.098 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:53.893+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.894838+0000) 2024-02-19T20:17:54.098 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:53.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.803950+0000) 2024-02-19T20:17:54.099 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:54.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:54.099 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:53.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:54.099 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:53.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:54.099 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:53.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:54.099 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:53.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:54.186+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.186922+0000) 2024-02-19T20:17:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:54.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:54.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:54.096+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:54.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:54.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:54.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:54.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:55.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:54.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.804034+0000) 2024-02-19T20:17:55.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:54.894+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.894982+0000) 2024-02-19T20:17:55.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:54.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:55.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:54.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:55.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:54.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:55.061 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:54.922+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:55.188+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.189514+0000) 2024-02-19T20:17:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:55.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:55.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:55.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:55.141+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:55.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:55.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:55.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:55.183+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:56.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:55.894+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.895164+0000) 2024-02-19T20:17:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:55.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:55.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:55.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:55.930+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:56.117 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:55.802+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.804146+0000) 2024-02-19T20:17:56.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:56.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:56.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:56.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:56.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:56.114+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:56.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:56.191+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.191811+0000) 2024-02-19T20:17:56.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:56.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:56.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:56.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:56.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:56.157+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:57.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:56.894+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.895368+0000) 2024-02-19T20:17:57.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:56.803+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.804327+0000) 2024-02-19T20:17:57.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:56.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:57.125 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:56.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:57.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:56.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:57.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:56.925+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:57.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:57.190+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.192142+0000) 2024-02-19T20:17:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:57.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:57.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:57.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:57.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:57.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:57.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:57.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:57.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:57.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:57.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:58.111 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:57.895+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.895539+0000) 2024-02-19T20:17:58.111 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:57.804+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.804522+0000) 2024-02-19T20:17:58.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:58.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:57.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:57.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:57.905+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:58.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:57.905+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:58.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:17:58.180+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55130 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:17:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:58.178+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.178563+0000) 2024-02-19T20:17:58.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:58.191+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.192399+0000) 2024-02-19T20:17:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:58.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:58.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:58.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:58.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:58.174+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.174524+0000) 2024-02-19T20:17:58.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:58.173+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.174162+0000) 2024-02-19T20:17:58.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:58.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:58.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:58.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:58.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:58.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:59.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:58.895+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.895731+0000) 2024-02-19T20:17:59.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:58.804+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.804697+0000) 2024-02-19T20:17:59.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:59.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:59.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:58.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:59.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:58.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:59.115 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:58.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:17:59.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:58.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:59.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:59.194+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.194982+0000) 2024-02-19T20:17:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:59.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:59.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:59.113+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:17:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:59.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:59.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:17:59.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:59.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:00.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:17:59.804+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.804795+0000) 2024-02-19T20:18:00.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:17:59.895+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.895901+0000) 2024-02-19T20:18:00.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:59.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:00.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:59.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:00.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:59.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:00.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:17:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:17:59.849+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:00.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:00.198+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.198774+0000) 2024-02-19T20:18:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:00.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:00.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:00.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:00.149+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:00.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:00.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:00.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:00.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:01.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:00.895+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.896102+0000) 2024-02-19T20:18:01.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:00.804+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.804944+0000) 2024-02-19T20:18:01.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:00.846+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:01.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:00.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:01.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:00.847+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:01.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:00.847+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:01.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:01.201+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.201863+0000) 2024-02-19T20:18:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:01.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:01.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:01.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:01.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:01.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:01.194+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:01.194+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:01.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:01.194+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:02.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:01.804+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.805101+0000) 2024-02-19T20:18:02.126 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:01.895+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.896298+0000) 2024-02-19T20:18:02.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:01.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:02.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:01.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:02.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:01.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:02.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:01.869+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:02.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:02.201+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.202108+0000) 2024-02-19T20:18:02.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:02.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:02.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:02.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:02.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:02.133+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:02.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:02.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:02.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:02.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:03.092 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:02.895+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.896414+0000) 2024-02-19T20:18:03.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:02.804+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.805284+0000) 2024-02-19T20:18:03.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:03.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:03.093 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:03.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:03.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:02.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:03.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:02.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:03.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:02.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:03.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:02.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:03.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:03.188+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55203 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:03.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:03.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:03.182+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.182658+0000) 2024-02-19T20:18:03.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:03.185+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.185677+0000) 2024-02-19T20:18:03.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:03.201+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.202321+0000) 2024-02-19T20:18:03.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:03.182+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.182526+0000) 2024-02-19T20:18:03.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:03.206+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:03.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:03.206+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:03.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:03.206+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:04.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:03.805+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.805491+0000) 2024-02-19T20:18:04.092 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:04.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:04.093 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:03.896+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.896641+0000) 2024-02-19T20:18:04.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:03.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:04.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:03.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:04.093 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:03.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:04.094 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:03.940+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:04.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:04.204+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.205199+0000) 2024-02-19T20:18:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:04.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:04.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:04.091+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:04.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:04.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:04.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:04.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:05.111 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:04.896+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.896795+0000) 2024-02-19T20:18:05.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:04.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:05.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:04.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:05.111 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:04.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:05.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:04.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:05.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:04.805+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.805635+0000) 2024-02-19T20:18:05.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:05.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:05.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:05.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:05.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:05.110+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:05.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:05.207+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.207984+0000) 2024-02-19T20:18:05.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:05.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:05.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:05.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:05.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:06.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:05.805+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.805842+0000) 2024-02-19T20:18:06.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:06.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:06.089 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:05.896+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.896978+0000) 2024-02-19T20:18:06.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:05.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:06.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:05.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:06.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:05.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:06.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:05.965+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:06.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:06.209+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.210177+0000) 2024-02-19T20:18:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:06.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:06.088+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:06.148+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:06.148+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:06.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:06.148+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:07.126 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:06.805+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.806021+0000) 2024-02-19T20:18:07.127 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:06.896+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.897102+0000) 2024-02-19T20:18:07.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:06.978+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:07.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:06.978+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:07.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:06.978+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:07.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:06.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:07.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:07.210+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.210471+0000) 2024-02-19T20:18:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:07.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:07.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:07.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:07.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:07.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:07.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:07.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:07.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:07.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:07.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:07.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:08.100 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:07.896+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.897231+0000) 2024-02-19T20:18:08.101 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:07.805+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.806189+0000) 2024-02-19T20:18:08.101 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:08.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:08.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:07.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:08.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:07.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:08.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:07.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:08.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:07.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:08.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:08.196+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55276 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:08.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:08.194+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.194757+0000) 2024-02-19T20:18:08.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:08.210+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.210693+0000) 2024-02-19T20:18:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:08.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:08.099+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:08.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:08.190+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.191307+0000) 2024-02-19T20:18:08.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:08.190+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.190711+0000) 2024-02-19T20:18:08.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:08.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:08.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:08.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:08.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:08.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:09.057 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:08.897+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.897407+0000) 2024-02-19T20:18:09.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:08.805+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.806380+0000) 2024-02-19T20:18:09.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:09.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:09.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:09.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:09.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:09.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:09.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:09.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:09.056+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:09.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.213+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.214144+0000) 2024-02-19T20:18:09.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:09.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:09.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:09.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:09.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:09.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:10.127 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:09.897+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.897571+0000) 2024-02-19T20:18:10.127 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:10.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:10.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:10.128 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:09.984+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:10.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:09.806+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.806493+0000) 2024-02-19T20:18:10.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:10.035+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:10.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:10.035+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:10.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:10.035+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:10.420 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:10.216+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.216799+0000) 2024-02-19T20:18:10.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:10.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:10.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:10.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:10.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:10.236+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:11.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:10.897+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.897779+0000) 2024-02-19T20:18:11.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:10.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:11.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:10.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:11.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:10.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:11.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:10.994+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:11.068 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:10.806+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.806621+0000) 2024-02-19T20:18:11.068 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:11.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:11.068 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:11.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:11.068 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:11.066+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:11.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:11.218+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.218920+0000) 2024-02-19T20:18:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:11.273+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:11.273+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:11.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:11.273+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:12.088 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:11.897+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.897941+0000) 2024-02-19T20:18:12.088 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:12.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:12.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:12.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.025+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:12.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:11.806+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.806813+0000) 2024-02-19T20:18:12.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:12.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:12.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:12.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:12.090 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:12.087+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:12.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.218+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.219175+0000) 2024-02-19T20:18:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:12.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:12.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:12.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:13.125 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:12.897+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.898105+0000) 2024-02-19T20:18:13.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:12.806+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.807021+0000) 2024-02-19T20:18:13.126 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:13.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:13.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:13.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:13.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:13.126 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:12.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:13.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:13.205+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55349 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:13.203+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.203473+0000) 2024-02-19T20:18:13.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:13.218+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.219409+0000) 2024-02-19T20:18:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:13.199+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.199517+0000) 2024-02-19T20:18:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:13.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:13.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:13.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:13.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:13.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:13.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:13.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:13.124+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:13.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:13.198+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.198946+0000) 2024-02-19T20:18:14.100 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:13.897+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.898324+0000) 2024-02-19T20:18:14.100 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:13.806+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.807194+0000) 2024-02-19T20:18:14.100 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:14.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:14.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:13.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:14.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:13.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:14.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:13.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:14.101 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:13.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:14.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:14.221+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.222295+0000) 2024-02-19T20:18:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:14.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:14.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:14.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:14.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:14.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:14.098+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:15.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:14.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.898493+0000) 2024-02-19T20:18:15.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:14.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.807371+0000) 2024-02-19T20:18:15.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:14.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:15.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:14.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:15.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:14.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:15.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:14.955+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:15.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:15.224+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.225189+0000) 2024-02-19T20:18:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:15.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:15.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:15.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:15.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:15.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:15.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:15.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:15.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:15.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:15.129+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:16.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:15.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.807575+0000) 2024-02-19T20:18:16.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:15.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.898685+0000) 2024-02-19T20:18:16.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:15.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:16.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:15.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:16.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:15.931+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:16.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:15.931+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:16.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:16.227+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.227318+0000) 2024-02-19T20:18:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:16.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:16.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:16.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:16.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:16.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:16.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:16.128+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:16.899 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:16.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.807731+0000) 2024-02-19T20:18:17.217 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:16.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.898882+0000) 2024-02-19T20:18:17.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:17.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:17.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:17.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:17.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:17.107+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:17.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:16.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:17.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:16.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:17.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:16.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:17.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:16.935+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:17.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:17.227+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.227570+0000) 2024-02-19T20:18:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:17.232+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:17.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:17.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:17.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:18.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:17.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.807910+0000) 2024-02-19T20:18:18.128 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:17.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.899034+0000) 2024-02-19T20:18:18.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:17.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:18.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:17.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:18.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:17.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:18.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:17.984+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:18.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:18.214+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55422 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:18.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:18.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:18.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:18.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:18.127+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:18.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:18.206+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.207119+0000) 2024-02-19T20:18:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:18.209+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.210215+0000) 2024-02-19T20:18:18.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:18.227+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.227768+0000) 2024-02-19T20:18:18.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:18.206+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.206898+0000) 2024-02-19T20:18:18.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:18.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:18.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:18.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:18.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:18.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:19.101 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:18.899+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.899208+0000) 2024-02-19T20:18:19.102 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:18.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.808107+0000) 2024-02-19T20:18:19.102 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:19.100+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:19.102 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:19.100+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:19.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:18.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:19.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:18.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:19.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:18.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:19.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:18.974+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:19.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:19.100+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:19.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:19.230+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.230147+0000) 2024-02-19T20:18:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:19.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:19.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:19.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:19.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:20.094 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:19.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.899351+0000) 2024-02-19T20:18:20.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:19.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.808263+0000) 2024-02-19T20:18:20.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:20.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:20.095 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:20.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:20.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:20.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:20.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.018+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:20.096 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.018+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:20.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.232+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.232914+0000) 2024-02-19T20:18:20.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:20.093+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:20.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:20.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:20.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:20.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:20.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:20.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:21.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:20.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.808476+0000) 2024-02-19T20:18:21.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:21.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:21.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:20.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.899545+0000) 2024-02-19T20:18:21.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:21.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:21.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.980+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:21.120 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:20.980+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:21.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:21.234+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.235957+0000) 2024-02-19T20:18:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:21.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:21.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:21.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:21.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:21.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:21.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:21.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:21.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:21.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:22.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:21.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.808645+0000) 2024-02-19T20:18:22.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:21.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.899708+0000) 2024-02-19T20:18:22.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:22.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:22.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:22.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:22.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:22.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:22.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:22.022+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:22.234+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.236238+0000) 2024-02-19T20:18:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:22.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:22.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:22.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:22.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:22.130+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:22.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:22.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:22.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:22.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:22.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:22.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:22.967 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:22.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.899853+0000) 2024-02-19T20:18:22.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:22.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.808852+0000) 2024-02-19T20:18:23.218 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:23.214+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.216404+0000) 2024-02-19T20:18:23.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:23.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:23.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:23.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:23.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:23.117+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:23.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:23.215+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.216694+0000) 2024-02-19T20:18:23.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.043+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:23.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.043+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:23.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.043+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:23.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.043+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:23.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:23.221+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55495 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:23.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.218+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.220255+0000) 2024-02-19T20:18:23.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.235+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.236472+0000) 2024-02-19T20:18:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:23.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:23.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:23.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:23.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:24.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:23.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.899974+0000) 2024-02-19T20:18:24.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:23.807+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.809054+0000) 2024-02-19T20:18:24.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:24.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:24.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:24.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:23.995+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:24.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:24.238+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.238989+0000) 2024-02-19T20:18:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:24.252+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:24.252+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:24.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:24.252+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:24.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:24.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:24.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:24.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:24.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:24.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:25.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:24.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.809184+0000) 2024-02-19T20:18:25.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:24.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.900091+0000) 2024-02-19T20:18:25.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:25.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:25.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:25.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:25.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:25.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:25.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:25.044+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:25.240+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.241868+0000) 2024-02-19T20:18:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:25.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:25.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:25.285+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:25.165+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:25.165+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:25.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:25.165+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:26.072 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:25.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.809343+0000) 2024-02-19T20:18:26.072 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:25.899+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.900248+0000) 2024-02-19T20:18:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:26.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:26.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:26.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:26.157+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:26.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:26.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:26.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:26.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:26.070+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:26.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:26.242+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.244070+0000) 2024-02-19T20:18:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:26.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:26.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:26.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:26.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:27.073 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:26.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.809516+0000) 2024-02-19T20:18:27.073 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:26.898+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.900448+0000) 2024-02-19T20:18:27.409 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:27.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:27.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:27.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:27.410 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:27.175+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:27.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:27.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:27.410 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:27.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:27.411 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:27.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:27.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:27.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:27.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:27.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:27.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:27.071+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:27.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:27.071+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:27.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:27.243+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.244330+0000) 2024-02-19T20:18:28.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:27.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.809669+0000) 2024-02-19T20:18:28.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:27.899+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.900563+0000) 2024-02-19T20:18:28.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:28.230+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55568 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:28.134+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:28.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:28.135+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:28.135+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:28.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:28.223+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.224753+0000) 2024-02-19T20:18:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:28.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:28.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:28.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:28.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:28.112+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:28.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:28.226+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.227695+0000) 2024-02-19T20:18:28.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:28.243+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.244572+0000) 2024-02-19T20:18:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:28.223+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.224831+0000) 2024-02-19T20:18:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:28.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:28.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:28.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:28.239+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:29.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:28.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.809864+0000) 2024-02-19T20:18:29.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:28.899+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.900766+0000) 2024-02-19T20:18:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:29.159+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:29.159+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:29.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:29.159+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:29.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:29.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:29.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:29.155+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:29.156+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:29.245+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.246828+0000) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:29.252+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:29.253+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:29.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:29.253+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:30.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:29.809+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.810049+0000) 2024-02-19T20:18:30.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:29.900+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.900935+0000) 2024-02-19T20:18:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:30.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:30.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:30.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:30.202+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:30.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:30.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:30.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:30.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:30.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:30.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:30.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:30.192+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:30.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:30.248+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.249488+0000) 2024-02-19T20:18:31.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:30.808+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.810271+0000) 2024-02-19T20:18:31.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:30.899+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.901101+0000) 2024-02-19T20:18:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:31.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:31.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:31.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:31.167+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:31.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:31.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:31.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:31.184+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:31.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:31.251+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.252750+0000) 2024-02-19T20:18:31.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:31.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:31.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:31.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:31.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:31.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:32.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:31.809+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.810463+0000) 2024-02-19T20:18:32.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:31.900+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.901227+0000) 2024-02-19T20:18:32.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:32.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:32.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:32.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:32.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:32.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:32.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:32.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:32.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:32.141+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:32.141+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:32.141+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:32.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:32.141+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:32.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:32.251+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.253007+0000) 2024-02-19T20:18:33.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:32.809+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.810552+0000) 2024-02-19T20:18:33.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:32.899+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.901350+0000) 2024-02-19T20:18:33.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:33.239+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55641 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:33.232+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.233572+0000) 2024-02-19T20:18:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:33.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:33.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:33.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:33.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:33.140+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:33.140+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:33.140+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:33.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:33.141+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:33.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:33.235+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.237079+0000) 2024-02-19T20:18:33.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:33.251+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.253270+0000) 2024-02-19T20:18:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:33.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:33.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:33.161+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:33.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:33.232+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.234093+0000) 2024-02-19T20:18:34.119 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:33.809+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.810738+0000) 2024-02-19T20:18:34.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:33.900+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.901506+0000) 2024-02-19T20:18:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:34.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:34.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:34.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:34.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:34.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:34.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:34.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:34.345+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:34.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:34.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:34.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:34.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:34.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:34.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:34.118+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:34.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:34.254+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.255586+0000) 2024-02-19T20:18:35.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:34.809+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.810821+0000) 2024-02-19T20:18:35.131 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:34.900+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.901646+0000) 2024-02-19T20:18:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:35.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:35.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:35.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:35.144+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:35.144+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:35.144+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:35.145+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:35.256+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.258161+0000) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:35.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:35.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:35.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:35.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:35.140+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:36.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:35.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.811000+0000) 2024-02-19T20:18:36.105 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:35.901+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.901845+0000) 2024-02-19T20:18:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:36.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:36.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:36.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:36.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:36.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:36.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:36.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:36.104+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:36.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:36.258+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.260316+0000) 2024-02-19T20:18:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:36.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:36.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:36.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:36.187+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:37.071 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:36.809+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.811106+0000) 2024-02-19T20:18:37.072 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:36.901+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.902004+0000) 2024-02-19T20:18:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:37.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:37.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:37.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:37.287+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:37.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:37.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:37.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:37.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:37.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:37.183+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:37.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:37.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:37.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:37.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:37.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:37.070+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:37.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:37.070+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:37.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:37.259+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.260634+0000) 2024-02-19T20:18:38.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:37.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.811269+0000) 2024-02-19T20:18:38.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:37.901+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.902183+0000) 2024-02-19T20:18:38.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:38.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:38.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:38.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:38.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:38.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:38.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:38.058+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:38.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:38.247+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55714 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:38.243+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.245264+0000) 2024-02-19T20:18:38.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:38.259+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.260932+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:38.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:38.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:38.217+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:38.240+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.242093+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:38.240+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.242220+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:38.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:38.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:38.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:38.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:38.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:39.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:38.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.811482+0000) 2024-02-19T20:18:39.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:38.901+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.902351+0000) 2024-02-19T20:18:39.132 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:39.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:39.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:39.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:39.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:39.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:39.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:39.041+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:39.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:39.263+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.263714+0000) 2024-02-19T20:18:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:39.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:39.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:39.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:39.188+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:39.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:39.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:39.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:39.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:40.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:39.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.811662+0000) 2024-02-19T20:18:40.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:39.901+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.902529+0000) 2024-02-19T20:18:40.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:40.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:40.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:40.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:40.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:40.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:40.137 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:40.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:40.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:40.265+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.266470+0000) 2024-02-19T20:18:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:40.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:40.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:40.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:40.143+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:40.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:40.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:40.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:40.296+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:41.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:40.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.811825+0000) 2024-02-19T20:18:41.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:40.902+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.902720+0000) 2024-02-19T20:18:41.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:41.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:41.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:41.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:41.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:41.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:41.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:41.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:41.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:41.268+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.268751+0000) 2024-02-19T20:18:41.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:41.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:41.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:41.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:41.184+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:41.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:41.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:41.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:41.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:42.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:41.902+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.902934+0000) 2024-02-19T20:18:42.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:41.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.812003+0000) 2024-02-19T20:18:42.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:42.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:42.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:42.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:42.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:42.016+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:42.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:42.016+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:42.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:42.268+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.269081+0000) 2024-02-19T20:18:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:42.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:42.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:42.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:42.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:42.153+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:42.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:42.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:42.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:42.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:42.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:42.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:43.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:42.810+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.812153+0000) 2024-02-19T20:18:43.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:42.902+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.903133+0000) 2024-02-19T20:18:43.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:43.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:43.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:43.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:43.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:43.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:43.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:43.012+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:43.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:43.255+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55788 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:43.249+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.250525+0000) 2024-02-19T20:18:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:43.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:43.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:43.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:43.302+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:43.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:43.253+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.254552+0000) 2024-02-19T20:18:43.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:43.267+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.269291+0000) 2024-02-19T20:18:43.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:43.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:43.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:43.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:43.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:43.139+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:43.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:43.249+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.250882+0000) 2024-02-19T20:18:44.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:43.811+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.812324+0000) 2024-02-19T20:18:44.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:43.902+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.903365+0000) 2024-02-19T20:18:44.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:44.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:44.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:44.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.271+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.271771+0000) 2024-02-19T20:18:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:44.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:44.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:44.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:44.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:44.138+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:44.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:44.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:44.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:45.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:44.811+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.812471+0000) 2024-02-19T20:18:45.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:44.902+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.903538+0000) 2024-02-19T20:18:45.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:45.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:45.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.970+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:45.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:44.970+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:45.274+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.274598+0000) 2024-02-19T20:18:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:45.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:45.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:45.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:45.148+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:45.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:45.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:45.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:46.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:45.903+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.903760+0000) 2024-02-19T20:18:46.134 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:45.811+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.812622+0000) 2024-02-19T20:18:46.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:45.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:46.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:45.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:46.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:45.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:46.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:45.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:46.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:46.276+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.276893+0000) 2024-02-19T20:18:46.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:46.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:46.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:46.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:46.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:46.132+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:46.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:46.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:46.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:46.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:47.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:46.811+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.812754+0000) 2024-02-19T20:18:47.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:46.903+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.903979+0000) 2024-02-19T20:18:47.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:46.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:47.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:46.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:47.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:46.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:47.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:46.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:47.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:47.276+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.277177+0000) 2024-02-19T20:18:47.388 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:47.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:47.388 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:47.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:47.388 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:47.171+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:47.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:47.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:47.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:48.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:47.812+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.812912+0000) 2024-02-19T20:18:48.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:47.903+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.904177+0000) 2024-02-19T20:18:48.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:47.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:48.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:47.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:48.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:47.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:48.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:47.929+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:48.385 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:48.264+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55860 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:48.386 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:48.256+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.258354+0000) 2024-02-19T20:18:48.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:48.260+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.261786+0000) 2024-02-19T20:18:48.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:48.276+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.277481+0000) 2024-02-19T20:18:48.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:48.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:48.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:48.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:48.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:48.194+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:48.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:48.257+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.258817+0000) 2024-02-19T20:18:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:48.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:48.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:48.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:49.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:48.812+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.813095+0000) 2024-02-19T20:18:49.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:48.903+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.904359+0000) 2024-02-19T20:18:49.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:48.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:49.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:48.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:49.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:48.938+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:49.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:48.939+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:49.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:49.279+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.280243+0000) 2024-02-19T20:18:49.438 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:49.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:49.439 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:49.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:49.439 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:49.207+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:49.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:49.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:49.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:49.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:50.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:49.812+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.813256+0000) 2024-02-19T20:18:50.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:49.904+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.904503+0000) 2024-02-19T20:18:50.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:49.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:50.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:49.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:50.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:49.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:50.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:49.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:50.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:50.282+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.283170+0000) 2024-02-19T20:18:50.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:50.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:50.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:50.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:50.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:50.226+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:50.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:50.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:50.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:51.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:50.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.813429+0000) 2024-02-19T20:18:51.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:50.904+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.904673+0000) 2024-02-19T20:18:51.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:50.943+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:51.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:50.943+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:51.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:50.943+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:51.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:50.943+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:51.449 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:51.285+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.285594+0000) 2024-02-19T20:18:51.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:51.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:51.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:51.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:51.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:51.193+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:51.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:51.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:51.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:51.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:52.134 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:51.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.813611+0000) 2024-02-19T20:18:52.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:51.904+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.904799+0000) 2024-02-19T20:18:52.134 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:51.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:52.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:51.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:52.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:51.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:52.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:51.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:52.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:52.285+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.285867+0000) 2024-02-19T20:18:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:52.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:52.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:52.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:52.239+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:52.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:52.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:52.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:52.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:52.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:52.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:53.134 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:52.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.813772+0000) 2024-02-19T20:18:53.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:52.904+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.904976+0000) 2024-02-19T20:18:53.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:53.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:53.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:53.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:53.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:53.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:53.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:53.031+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:53.395 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:53.272+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 55934 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:53.396 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:53.266+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.267172+0000) 2024-02-19T20:18:53.396 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:53.270+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.270865+0000) 2024-02-19T20:18:53.396 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:53.285+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.286078+0000) 2024-02-19T20:18:53.397 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:53.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:53.397 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:53.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:53.397 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:53.214+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:53.397 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:53.267+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.267876+0000) 2024-02-19T20:18:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:53.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:53.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:53.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:53.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:54.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:53.904+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.905164+0000) 2024-02-19T20:18:54.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:53.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.813906+0000) 2024-02-19T20:18:54.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:54.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:54.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:54.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:54.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:54.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:54.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:54.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:54.287+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.288364+0000) 2024-02-19T20:18:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:54.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:54.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:54.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:54.206+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:54.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:54.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:54.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:54.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:54.815 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:54.490+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:55.134 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:54.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.814053+0000) 2024-02-19T20:18:55.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:54.904+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.905335+0000) 2024-02-19T20:18:55.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:55.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:55.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:55.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:55.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:55.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:55.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:55.023+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:55.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:55.290+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.291237+0000) 2024-02-19T20:18:55.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:55.190+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:55.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:55.190+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:55.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:55.190+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:55.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:55.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:55.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:55.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:56.134 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:55.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.814234+0000) 2024-02-19T20:18:56.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:55.905+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.905541+0000) 2024-02-19T20:18:56.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:56.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:56.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:56.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:56.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:56.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:56.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:56.011+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:56.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:56.293+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.293511+0000) 2024-02-19T20:18:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:56.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:56.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:56.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:56.213+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:56.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:56.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:56.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:56.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:57.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:56.813+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.814383+0000) 2024-02-19T20:18:57.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:56.905+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.905665+0000) 2024-02-19T20:18:57.135 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:57.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:57.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:57.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:57.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:57.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:57.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:57.015+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:57.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:57.293+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.293782+0000) 2024-02-19T20:18:57.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:57.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:57.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:57.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:57.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:57.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:57.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:57.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:57.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:57.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:57.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:57.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:58.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:57.814+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.814585+0000) 2024-02-19T20:18:58.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:57.905+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.905848+0000) 2024-02-19T20:18:58.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:58.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:58.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:58.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:58.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:58.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:58.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:58.011+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:58.405 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:18:58.280+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56006 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:18:58.406 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:58.275+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.275542+0000) 2024-02-19T20:18:58.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:58.278+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.278658+0000) 2024-02-19T20:18:58.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:58.293+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.293996+0000) 2024-02-19T20:18:58.407 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:58.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:58.407 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:58.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:58.407 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:58.208+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:58.407 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:58.274+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.274822+0000) 2024-02-19T20:18:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:58.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:58.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:58.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:58.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:59.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:58.814+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.814805+0000) 2024-02-19T20:18:59.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:58.905+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.906033+0000) 2024-02-19T20:18:59.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:59.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:59.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:59.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:59.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:59.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:18:59.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:59.044+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:59.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:18:59.296+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.296530+0000) 2024-02-19T20:18:59.431 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:59.253+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:59.431 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:59.253+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:59.431 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:59.253+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:18:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:59.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:59.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:59.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:18:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:59.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:00.091 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:18:59.814+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.814958+0000) 2024-02-19T20:19:00.091 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:18:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:18:59.905+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.906196+0000) 2024-02-19T20:19:00.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:00.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:00.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:00.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:00.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:00.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:00.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:00.090+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:00.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:00.090+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:00.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:00.090+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:00.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:00.090+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:00.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:00.298+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.299287+0000) 2024-02-19T20:19:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:00.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:00.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:00.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:01.112 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:00.814+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.815141+0000) 2024-02-19T20:19:01.113 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:00.905+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.906411+0000) 2024-02-19T20:19:01.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:01.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:01.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:01.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:01.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:01.300+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:01.432 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:01.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:01.433 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:01.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:01.433 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:01.112+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:01.433 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:01.112+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:01.433 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:01.301+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.301515+0000) 2024-02-19T20:19:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:01.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:01.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:01.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:01.431+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:02.076 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:01.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.815354+0000) 2024-02-19T20:19:02.077 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:01.906+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.906591+0000) 2024-02-19T20:19:02.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:02.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:02.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:02.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:02.339 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:02.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:02.339 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:02.075+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:02.339 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:02.301+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.301837+0000) 2024-02-19T20:19:02.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:02.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:02.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:02.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:02.644 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:02.337+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:02.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:02.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:02.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:02.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:02.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:02.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:02.907 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:02.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.815539+0000) 2024-02-19T20:19:03.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:02.906+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.906814+0000) 2024-02-19T20:19:03.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:03.097+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:03.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:03.097+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:03.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:03.097+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:03.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:03.098+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:03.282+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.283169+0000) 2024-02-19T20:19:03.478 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:03.289+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56080 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:03.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:03.286+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.286633+0000) 2024-02-19T20:19:03.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:03.301+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.302036+0000) 2024-02-19T20:19:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:03.282+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.282989+0000) 2024-02-19T20:19:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:03.321+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:03.321+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:03.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:03.321+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:03.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:03.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:03.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:03.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:03.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:03.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:04.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:03.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.815696+0000) 2024-02-19T20:19:04.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:03.906+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.906956+0000) 2024-02-19T20:19:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:04.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:04.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:04.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:04.365+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:04.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:04.139+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:04.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:04.139+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:04.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:04.139+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:04.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:04.139+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:04.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:04.304+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.304668+0000) 2024-02-19T20:19:04.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:04.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:04.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:04.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:04.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:04.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:05.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:04.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.815831+0000) 2024-02-19T20:19:05.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:04.906+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.907148+0000) 2024-02-19T20:19:05.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:05.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:05.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:05.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:05.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:05.163+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:05.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:05.163+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:05.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:05.307+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.307473+0000) 2024-02-19T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:05.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:05.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:05.716 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:05.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:05.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:05.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:05.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:05.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:05.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:05.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:05.975 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:05.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.816014+0000) 2024-02-19T20:19:05.975 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:05.906+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.907350+0000) 2024-02-19T20:19:06.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:06.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:06.225 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:06.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:06.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:06.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:06.226 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:06.164+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:06.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:06.309+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.309903+0000) 2024-02-19T20:19:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:06.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:06.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:06.478 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:06.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:06.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:06.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:06.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:06.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:06.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:06.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:07.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:06.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.816167+0000) 2024-02-19T20:19:07.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:06.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.907510+0000) 2024-02-19T20:19:07.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:07.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:07.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:07.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:07.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:07.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:07.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:07.206+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:07.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:07.310+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.310238+0000) 2024-02-19T20:19:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:07.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:07.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:07.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:07.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:07.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:07.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:07.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:08.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:07.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.816339+0000) 2024-02-19T20:19:08.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:07.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.907687+0000) 2024-02-19T20:19:08.415 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:08.298+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56152 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:08.415 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:08.291+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.291492+0000) 2024-02-19T20:19:08.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:08.290+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.291377+0000) 2024-02-19T20:19:08.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:08.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:08.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:08.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:08.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:08.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:08.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:08.199+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:08.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:08.199+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:08.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:08.199+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:08.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:08.199+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:08.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:08.295+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.295476+0000) 2024-02-19T20:19:08.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:08.310+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.310471+0000) 2024-02-19T20:19:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:08.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:08.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:08.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:08.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:09.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:08.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.816543+0000) 2024-02-19T20:19:09.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:08.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.907893+0000) 2024-02-19T20:19:09.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:09.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:09.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:09.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:09.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:09.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:09.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:09.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:09.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:09.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:09.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:09.161+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:09.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:09.161+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:09.451 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:09.313+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.313521+0000) 2024-02-19T20:19:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:09.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:09.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:09.449+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:10.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:09.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.816688+0000) 2024-02-19T20:19:10.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:09.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.908054+0000) 2024-02-19T20:19:10.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:10.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:10.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:10.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:10.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:10.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:10.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:10.170+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:10.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:10.315+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.316197+0000) 2024-02-19T20:19:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:10.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:10.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:10.398+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:10.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:10.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:10.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:10.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:11.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:10.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.816875+0000) 2024-02-19T20:19:11.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:10.908+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.908243+0000) 2024-02-19T20:19:11.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:11.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:11.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:11.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:11.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:11.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:11.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:11.193+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:11.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:11.317+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.318585+0000) 2024-02-19T20:19:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:11.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:11.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:11.409+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:11.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:11.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:11.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:11.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:11.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:12.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:11.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.817081+0000) 2024-02-19T20:19:12.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:11.906+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.908427+0000) 2024-02-19T20:19:12.424 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:12.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:12.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:12.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:12.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:12.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:12.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:12.186+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:12.425 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:12.317+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.318936+0000) 2024-02-19T20:19:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:12.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:12.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:12.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:12.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:12.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:12.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:13.137 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:12.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.817186+0000) 2024-02-19T20:19:13.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:12.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.908652+0000) 2024-02-19T20:19:13.426 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:13.305+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56226 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:13.426 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:13.298+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.300393+0000) 2024-02-19T20:19:13.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:13.299+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.300620+0000) 2024-02-19T20:19:13.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:13.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:13.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:13.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:13.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:13.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:13.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:13.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:13.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:13.303+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.304843+0000) 2024-02-19T20:19:13.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:13.317+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.319236+0000) 2024-02-19T20:19:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:13.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:13.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:13.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:13.444+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:13.444+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:13.444+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:14.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:13.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.908855+0000) 2024-02-19T20:19:14.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:13.815+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.817356+0000) 2024-02-19T20:19:14.398 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:14.211+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:14.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:14.211+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:14.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:14.211+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:14.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:14.211+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:14.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:14.320+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.321671+0000) 2024-02-19T20:19:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:14.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:14.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:14.458+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:14.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:14.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:14.397+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:15.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:14.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.817512+0000) 2024-02-19T20:19:15.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:14.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.908972+0000) 2024-02-19T20:19:15.464 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:15.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:15.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:15.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:15.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:15.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:15.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:15.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:15.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:15.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:15.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:15.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:15.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:15.219+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:15.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:15.323+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.324366+0000) 2024-02-19T20:19:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:15.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:15.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:15.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:16.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:15.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.817687+0000) 2024-02-19T20:19:16.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:15.908+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.909152+0000) 2024-02-19T20:19:16.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:16.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:16.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:16.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:16.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:16.191+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:16.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:16.191+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:16.465 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:16.325+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.326553+0000) 2024-02-19T20:19:16.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:16.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:16.466 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:16.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:16.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:16.463+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:17.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:16.816+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.817835+0000) 2024-02-19T20:19:17.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:16.907+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.909353+0000) 2024-02-19T20:19:17.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:17.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:17.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:17.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:17.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:17.147+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:17.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:17.147+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:17.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:17.325+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.326805+0000) 2024-02-19T20:19:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:17.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:17.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:17.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:17.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:17.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:17.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:18.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:17.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.817997+0000) 2024-02-19T20:19:18.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:17.908+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.909551+0000) 2024-02-19T20:19:18.440 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:18.314+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56298 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:18.441 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:18.307+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.308709+0000) 2024-02-19T20:19:18.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:18.307+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.309052+0000) 2024-02-19T20:19:18.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:18.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:18.441 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:18.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:18.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:18.185+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:18.185+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:18.185+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:18.185+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:18.311+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.312547+0000) 2024-02-19T20:19:18.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:18.325+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.327061+0000) 2024-02-19T20:19:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:18.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:18.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:18.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:18.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:19.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:18.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.818186+0000) 2024-02-19T20:19:19.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:18.908+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.909756+0000) 2024-02-19T20:19:19.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:19.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:19.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:19.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:19.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:19.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:19.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:19.179+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:19.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:19.179+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:19.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:19.179+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:19.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:19.179+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:19.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:19.328+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.329745+0000) 2024-02-19T20:19:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:19.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:19.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:19.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:19.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:20.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:19.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.818371+0000) 2024-02-19T20:19:20.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:19.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.909941+0000) 2024-02-19T20:19:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:20.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:20.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:20.317+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:20.211+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:20.211+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:20.211+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:20.211+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:20.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:20.330+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.332408+0000) 2024-02-19T20:19:20.818 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:20.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:20.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:20.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:20.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:20.502+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:21.076 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:20.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.818468+0000) 2024-02-19T20:19:21.076 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:20.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.910130+0000) 2024-02-19T20:19:21.335 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:21.237+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:21.335 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:21.237+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:21.335 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:21.237+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:21.336 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:21.237+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:21.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:21.334+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.334901+0000) 2024-02-19T20:19:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:21.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:21.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:21.469+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:21.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:21.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:21.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:22.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:21.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.818653+0000) 2024-02-19T20:19:22.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:21.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.910343+0000) 2024-02-19T20:19:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:22.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:22.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:22.366+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:22.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:22.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:22.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:22.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:22.277+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:22.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:22.334+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.335228+0000) 2024-02-19T20:19:22.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:22.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:22.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:22.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:22.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:22.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:23.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:22.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.818800+0000) 2024-02-19T20:19:23.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:22.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.910468+0000) 2024-02-19T20:19:23.444 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:23.323+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56373 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:23.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:23.316+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.318033+0000) 2024-02-19T20:19:23.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:23.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:23.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:23.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:23.445 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:23.442+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:23.273+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:23.273+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:23.273+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:23.273+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:23.320+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.322137+0000) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:23.334+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.335473+0000) 2024-02-19T20:19:23.446 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:23.316+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.317810+0000) 2024-02-19T20:19:23.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:23.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:23.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:23.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:23.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:23.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:24.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:23.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.818922+0000) 2024-02-19T20:19:24.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:23.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.910604+0000) 2024-02-19T20:19:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:24.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:24.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:24.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:24.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:24.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:24.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:24.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:24.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:24.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:24.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:24.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:24.281+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:24.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:24.337+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.338913+0000) 2024-02-19T20:19:24.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:24.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:24.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:24.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:24.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:24.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:24.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:24.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.819056+0000) 2024-02-19T20:19:25.139 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:24.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.910765+0000) 2024-02-19T20:19:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:25.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:25.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:25.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:25.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:25.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:25.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:25.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:25.265+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:25.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:25.340+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.341515+0000) 2024-02-19T20:19:25.819 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:25.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:25.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:25.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:25.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:25.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:25.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:25.817+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.819275+0000) 2024-02-19T20:19:26.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:25.910+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.910975+0000) 2024-02-19T20:19:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:26.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:26.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:26.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:26.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:26.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:26.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:26.231+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:26.481 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:26.231+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:26.482 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:26.232+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:26.232+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:26.485 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:26.342+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.343904+0000) 2024-02-19T20:19:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:26.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:26.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:26.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:26.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:26.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:27.140 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:26.818+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.819515+0000) 2024-02-19T20:19:27.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:26.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.911171+0000) 2024-02-19T20:19:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:27.331+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:27.331+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:27.331+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:27.201+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:27.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:27.201+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:27.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:27.201+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:27.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:27.201+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:27.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:27.343+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.344161+0000) 2024-02-19T20:19:27.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:27.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:27.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:27.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:27.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:27.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:28.140 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:27.819+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.819728+0000) 2024-02-19T20:19:28.141 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:27.909+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.911328+0000) 2024-02-19T20:19:28.455 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:28.330+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56445 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:28.455 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:28.324+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.325879+0000) 2024-02-19T20:19:28.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:28.324+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.325789+0000) 2024-02-19T20:19:28.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:28.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:28.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:28.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:28.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:28.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:28.456 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:28.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:28.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:28.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:28.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:28.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:28.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:28.177+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:28.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:28.327+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.328724+0000) 2024-02-19T20:19:28.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:28.342+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.344413+0000) 2024-02-19T20:19:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:28.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:28.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:28.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:29.140 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:28.818+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.819868+0000) 2024-02-19T20:19:29.141 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:28.910+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.911503+0000) 2024-02-19T20:19:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:29.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:29.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:29.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:29.380+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:29.151+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:29.151+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:29.151+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:29.151+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:29.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:29.346+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.347806+0000) 2024-02-19T20:19:29.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:29.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:29.820 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:29.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:29.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:29.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:30.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:29.818+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.820044+0000) 2024-02-19T20:19:30.141 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:29.911+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.911674+0000) 2024-02-19T20:19:30.421 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:30.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:30.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:30.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:30.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:30.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:30.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:30.154+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:30.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:30.348+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.350390+0000) 2024-02-19T20:19:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:30.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:30.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:30.420+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:30.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:30.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:30.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:30.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:30.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:30.509+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:31.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:30.911+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.911885+0000) 2024-02-19T20:19:31.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:30.818+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.820250+0000) 2024-02-19T20:19:31.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:31.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:31.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:31.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:31.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:31.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:31.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:31.178+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:31.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:31.351+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.352652+0000) 2024-02-19T20:19:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:31.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:31.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:31.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:31.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:31.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:31.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:31.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:31.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:31.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:31.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:32.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:31.819+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.820441+0000) 2024-02-19T20:19:32.141 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:31.910+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.912048+0000) 2024-02-19T20:19:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:32.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:32.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:32.382+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:32.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:32.157+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:32.157+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:32.157+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:32.157+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:32.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:32.352+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.352903+0000) 2024-02-19T20:19:32.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:32.563+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:32.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:32.563+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:32.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:32.563+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:33.140 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:32.820+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.820685+0000) 2024-02-19T20:19:33.140 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:32.911+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.912214+0000) 2024-02-19T20:19:33.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:33.338+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56519 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:33.391 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:33.333+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.334561+0000) 2024-02-19T20:19:33.392 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:33.332+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.334058+0000) 2024-02-19T20:19:33.392 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:33.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:33.392 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:33.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:33.392 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:33.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:33.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:33.139+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:33.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:33.139+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:33.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:33.139+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:33.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:33.139+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:33.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:33.335+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.337149+0000) 2024-02-19T20:19:33.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:33.351+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.353196+0000) 2024-02-19T20:19:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:33.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:33.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:33.524+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:34.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:33.819+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.820884+0000) 2024-02-19T20:19:34.141 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:33.911+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.912373+0000) 2024-02-19T20:19:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:34.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:34.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:34.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:34.369+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:34.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:34.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:34.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:34.175+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:34.175+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:34.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:34.355+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.355586+0000) 2024-02-19T20:19:34.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:34.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:34.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:34.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:34.821 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:34.518+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:35.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:34.819+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.821020+0000) 2024-02-19T20:19:35.141 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:34.911+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.912524+0000) 2024-02-19T20:19:35.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:35.141+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:35.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:35.141+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:35.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:35.141+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:35.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:35.141+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:35.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:35.356+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.358194+0000) 2024-02-19T20:19:35.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:35.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:35.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:35.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:35.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:35.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:35.511+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:36.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:35.820+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.821122+0000) 2024-02-19T20:19:36.142 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:35.912+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.912741+0000) 2024-02-19T20:19:36.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:36.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:36.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:36.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:36.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:36.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:36.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:36.168+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:36.437 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:36.360+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.360696+0000) 2024-02-19T20:19:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:36.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:36.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:36.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:36.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:36.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:36.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:36.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:36.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:36.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:36.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:37.141 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:36.819+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.821224+0000) 2024-02-19T20:19:37.142 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:36.911+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.912891+0000) 2024-02-19T20:19:37.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:37.186+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:37.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:37.187+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:37.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:37.187+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:37.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:37.187+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:37.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:37.360+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.361066+0000) 2024-02-19T20:19:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:37.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:37.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:37.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:37.556+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:37.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:37.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:37.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:37.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:37.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:38.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:37.820+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.821402+0000) 2024-02-19T20:19:38.142 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:37.912+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.913096+0000) 2024-02-19T20:19:38.452 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:38.347+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56591 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:38.452 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:38.339+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.341331+0000) 2024-02-19T20:19:38.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:38.340+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.341605+0000) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:38.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:38.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:38.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:38.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:38.200+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:38.343+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.345070+0000) 2024-02-19T20:19:38.453 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:38.359+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.361307+0000) 2024-02-19T20:19:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:38.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:38.450+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:38.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:38.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:38.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:39.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:38.821+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.821568+0000) 2024-02-19T20:19:39.142 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:38.912+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.913292+0000) 2024-02-19T20:19:39.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:39.153+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:39.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:39.153+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:39.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:39.153+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:39.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:39.153+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:39.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:39.363+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.363737+0000) 2024-02-19T20:19:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:39.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:39.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:39.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:39.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:39.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:39.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:40.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:39.821+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.821743+0000) 2024-02-19T20:19:40.142 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:39.913+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.913471+0000) 2024-02-19T20:19:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:40.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:40.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:40.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:40.166+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:40.365+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.366393+0000) 2024-02-19T20:19:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:40.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:40.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:40.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:40.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:40.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:40.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:40.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:41.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:40.821+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.821907+0000) 2024-02-19T20:19:41.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:40.913+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.913603+0000) 2024-02-19T20:19:41.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:41.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:41.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:41.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:41.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:41.181+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:41.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:41.181+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:41.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:41.368+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.368863+0000) 2024-02-19T20:19:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:41.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:41.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:41.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:41.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:41.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:41.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:41.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:41.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:42.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:41.821+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.822081+0000) 2024-02-19T20:19:42.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:41.913+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.913784+0000) 2024-02-19T20:19:42.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:42.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:42.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:42.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:42.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:42.200+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:42.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:42.200+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:42.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:42.368+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.369175+0000) 2024-02-19T20:19:42.708 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:42.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:42.709 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:42.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:42.709 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:42.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:42.913+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.913930+0000) 2024-02-19T20:19:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:42.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:42.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:42.707+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:42.821+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.822267+0000) 2024-02-19T20:19:43.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:43.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:43.352 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:43.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:43.352 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:43.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:43.352 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:43.183+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:43.659 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:43.356+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56665 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:43.659 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:43.354+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.354558+0000) 2024-02-19T20:19:43.659 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:43.368+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.369362+0000) 2024-02-19T20:19:43.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:43.350+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.350559+0000) 2024-02-19T20:19:43.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:43.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:43.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:43.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:43.660 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:43.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:43.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:43.350+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.350668+0000) 2024-02-19T20:19:43.660 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:43.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:43.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:43.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:43.661 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:43.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:43.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:43.821+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.822393+0000) 2024-02-19T20:19:44.214 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:43.913+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.914128+0000) 2024-02-19T20:19:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:44.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:44.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:44.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:44.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:44.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:44.371+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.372020+0000) 2024-02-19T20:19:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:44.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:44.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:44.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:44.370+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:44.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:44.622+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:44.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:44.622+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:44.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:44.622+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:44.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:44.822+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.822538+0000) 2024-02-19T20:19:45.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:44.913+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.914273+0000) 2024-02-19T20:19:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:45.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:45.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:45.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:45.177+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:45.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:45.177+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:45.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:45.374+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.374626+0000) 2024-02-19T20:19:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:45.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:45.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:45.479 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:45.413+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:45.914 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:45.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:45.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:45.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:45.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:45.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:45.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:45.822+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.822712+0000) 2024-02-19T20:19:46.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:45.914+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.914480+0000) 2024-02-19T20:19:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:46.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:46.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:46.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:46.159+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:46.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:46.159+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:46.603 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:46.376+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.377304+0000) 2024-02-19T20:19:46.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:46.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:46.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:46.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:46.603 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:46.409+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:46.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:46.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:46.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:46.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:46.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:46.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:46.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:46.822+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.822905+0000) 2024-02-19T20:19:47.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:46.914+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.914648+0000) 2024-02-19T20:19:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:47.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:47.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:47.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:47.146+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:47.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:47.146+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:47.573 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:47.377+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.377622+0000) 2024-02-19T20:19:47.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:47.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:47.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:47.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:47.573 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:47.386+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:47.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:47.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:47.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:47.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:47.823 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:47.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:48.079 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:47.822+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.823083+0000) 2024-02-19T20:19:48.080 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:47.914+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.914839+0000) 2024-02-19T20:19:48.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:48.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:48.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:48.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:48.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:48.166+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:48.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:48.166+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:48.610 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:48.364+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56737 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:48.610 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:48.362+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.363369+0000) 2024-02-19T20:19:48.610 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:48.377+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.377845+0000) 2024-02-19T20:19:48.611 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:48.358+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.358612+0000) 2024-02-19T20:19:48.611 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:48.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:48.611 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:48.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:48.611 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:48.609+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:48.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:48.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:48.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:48.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:48.612 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:48.341+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:48.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:48.358+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.359032+0000) 2024-02-19T20:19:48.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:48.822+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.823288+0000) 2024-02-19T20:19:49.228 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:48.914+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.915028+0000) 2024-02-19T20:19:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:49.158+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:49.158+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:49.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:49.158+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:49.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:49.158+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:49.613 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:49.381+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.381469+0000) 2024-02-19T20:19:49.613 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:49.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:49.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:49.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:49.614 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:49.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:49.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:49.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:49.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:49.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:49.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:49.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:49.916 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:49.823+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.823468+0000) 2024-02-19T20:19:50.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:49.914+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.915181+0000) 2024-02-19T20:19:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:50.196+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:50.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:50.196+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:50.196+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:50.196+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:50.479 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:50.383+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.384280+0000) 2024-02-19T20:19:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:50.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:50.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:50.480 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:50.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:50.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:50.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:50.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:50.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:50.916 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:50.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:50.916 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:50.823+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.823693+0000) 2024-02-19T20:19:51.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:50.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.915369+0000) 2024-02-19T20:19:51.569 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:51.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:51.569 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:51.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:51.569 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:51.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:51.570 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:51.243+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:51.570 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:51.243+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:51.570 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:51.243+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:51.570 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:51.243+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:51.570 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:51.386+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.386838+0000) 2024-02-19T20:19:51.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:51.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:51.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:51.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:51.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:51.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:52.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:51.823+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.823900+0000) 2024-02-19T20:19:52.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:51.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.915565+0000) 2024-02-19T20:19:52.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:52.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:52.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:52.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:52.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:52.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:52.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:52.227+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:52.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:52.386+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.387118+0000) 2024-02-19T20:19:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:52.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:52.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:52.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:52.592+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:52.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:52.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:52.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:52.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:53.121 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:52.823+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.824085+0000) 2024-02-19T20:19:53.121 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:52.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.915740+0000) 2024-02-19T20:19:53.371 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:53.366+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.367296+0000) 2024-02-19T20:19:53.372 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:53.366+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.366754+0000) 2024-02-19T20:19:53.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:53.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:53.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:53.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:53.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:53.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:53.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:53.261+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:53.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:53.370+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.371202+0000) 2024-02-19T20:19:53.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:53.373+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56811 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:53.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:53.386+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.387369+0000) 2024-02-19T20:19:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:53.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:53.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:53.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:53.559+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:53.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:53.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:53.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:53.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:53.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:53.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:54.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:53.823+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.824253+0000) 2024-02-19T20:19:54.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:53.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.915941+0000) 2024-02-19T20:19:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:54.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:54.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:54.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:54.272+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:54.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:54.389+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.389693+0000) 2024-02-19T20:19:54.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:54.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:54.825 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:54.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:54.825 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:54.568+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:54.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:54.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:54.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:54.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:54.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:54.489+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:55.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:54.823+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.824426+0000) 2024-02-19T20:19:55.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:54.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.916119+0000) 2024-02-19T20:19:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:55.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:55.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:55.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:55.272+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:55.478 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:55.392+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.392553+0000) 2024-02-19T20:19:55.825 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:55.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:55.825 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:55.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:55.825 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:55.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:55.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:55.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:55.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:55.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:55.826 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:55.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:56.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:55.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.824646+0000) 2024-02-19T20:19:56.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:55.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.916235+0000) 2024-02-19T20:19:56.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:56.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:56.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:56.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:56.396 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:56.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:56.396 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:56.235+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:56.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:56.394+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.395055+0000) 2024-02-19T20:19:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:56.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:56.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:56.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:56.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:56.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:56.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:56.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:56.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:56.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:56.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:57.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:56.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.824827+0000) 2024-02-19T20:19:57.145 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:56.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.916426+0000) 2024-02-19T20:19:57.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:57.246+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:57.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:57.246+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:57.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:57.246+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:57.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:57.246+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:57.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:57.395+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.395370+0000) 2024-02-19T20:19:57.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:57.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:57.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:57.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:57.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:57.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:57.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:57.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:57.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:57.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:57.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:57.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:58.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:57.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.825027+0000) 2024-02-19T20:19:58.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:57.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.916572+0000) 2024-02-19T20:19:58.395 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:58.374+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.375281+0000) 2024-02-19T20:19:58.396 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:19:58.382+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56883 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:19:58.396 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:58.374+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.375056+0000) 2024-02-19T20:19:58.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:58.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:58.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:58.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:58.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:58.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:58.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:58.209+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:58.397 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:58.377+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.377976+0000) 2024-02-19T20:19:58.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:58.395+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.395604+0000) 2024-02-19T20:19:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:58.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:58.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:58.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:58.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:58.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:58.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:58.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:59.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:58.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.825188+0000) 2024-02-19T20:19:59.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:58.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.916743+0000) 2024-02-19T20:19:59.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:59.234+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:59.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:59.234+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:59.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:59.234+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:19:59.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:59.234+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:59.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:19:59.398+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.398785+0000) 2024-02-19T20:19:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:59.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:59.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:59.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:59.474+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:19:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:59.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:59.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:19:59.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:59.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:00.145 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:19:59.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.825367+0000) 2024-02-19T20:20:00.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:19:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:19:59.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.916879+0000) 2024-02-19T20:20:00.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:00.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:00.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:00.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:00.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:00.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:00.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:00.281+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:00.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:00.401+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.401763+0000) 2024-02-19T20:20:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:00.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:00.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:00.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:00.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:00.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:00.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:00.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:00.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:00.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:00.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:00.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:01.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:00.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.825574+0000) 2024-02-19T20:20:01.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:00.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.917004+0000) 2024-02-19T20:20:01.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:01.239+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:01.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:01.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:01.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:01.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:01.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:01.240+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:01.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:01.404+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.404623+0000) 2024-02-19T20:20:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:01.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:01.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:01.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:01.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:01.507+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:01.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:01.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:01.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:01.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:01.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:01.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:02.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:01.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.825757+0000) 2024-02-19T20:20:02.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:01.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.917180+0000) 2024-02-19T20:20:02.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:02.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:02.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:02.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:02.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:02.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:02.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:02.250+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:02.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:02.403+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.404928+0000) 2024-02-19T20:20:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:02.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:02.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:02.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:02.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:02.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:02.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:02.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:03.146 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:02.915+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.917356+0000) 2024-02-19T20:20:03.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:02.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.826019+0000) 2024-02-19T20:20:03.400 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:03.390+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 56957 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:03.400 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:03.382+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.384313+0000) 2024-02-19T20:20:03.401 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:03.382+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.384201+0000) 2024-02-19T20:20:03.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:03.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:03.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:03.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:03.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:03.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:03.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:03.217+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:03.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:03.386+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.388232+0000) 2024-02-19T20:20:03.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:03.403+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.405178+0000) 2024-02-19T20:20:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:03.537+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:03.537+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:03.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:03.537+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:03.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:03.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:03.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:03.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:03.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:03.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:03.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.917529+0000) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:03.824+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.826222+0000) 2024-02-19T20:20:04.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:04.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:04.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:04.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:04.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:04.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:04.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:04.170+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:04.406+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.407636+0000) 2024-02-19T20:20:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:04.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:04.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:04.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:04.510+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:04.569+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:04.569+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:04.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:04.569+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:05.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:04.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.826355+0000) 2024-02-19T20:20:05.147 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:04.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.917684+0000) 2024-02-19T20:20:05.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:05.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:05.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:05.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:05.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:05.212+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:05.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:05.212+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:05.409+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.410370+0000) 2024-02-19T20:20:05.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:05.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:05.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:05.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:05.516+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:05.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:05.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:05.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:05.557+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:06.146 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:05.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.826519+0000) 2024-02-19T20:20:06.147 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:05.916+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.917884+0000) 2024-02-19T20:20:06.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:06.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:06.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:06.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:06.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:06.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:06.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:06.170+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:06.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:06.411+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.412812+0000) 2024-02-19T20:20:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:06.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:06.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:06.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:06.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:06.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:06.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:06.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:06.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:07.131 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:06.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.826733+0000) 2024-02-19T20:20:07.132 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:06.917+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.918068+0000) 2024-02-19T20:20:07.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:07.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:07.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:07.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:07.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:07.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:07.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:07.130+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:07.411+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.413150+0000) 2024-02-19T20:20:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:07.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:07.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:07.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:07.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:07.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:07.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:07.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:07.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:07.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:08.132 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:07.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.826899+0000) 2024-02-19T20:20:08.133 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:07.917+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.918271+0000) 2024-02-19T20:20:08.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:08.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:08.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:08.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:08.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:08.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:08.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:08.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:08.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:08.399+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57029 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:08.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:08.395+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.396933+0000) 2024-02-19T20:20:08.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:08.411+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.413360+0000) 2024-02-19T20:20:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:08.392+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.393725+0000) 2024-02-19T20:20:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:08.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:08.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:08.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:08.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:08.391+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.393392+0000) 2024-02-19T20:20:08.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:08.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:08.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:08.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:08.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:08.587+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:09.147 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:08.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.827048+0000) 2024-02-19T20:20:09.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:08.917+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.918473+0000) 2024-02-19T20:20:09.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:09.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:09.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:09.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:09.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:09.156+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:09.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:09.156+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:09.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:09.415+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.416840+0000) 2024-02-19T20:20:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:09.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:09.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:09.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:09.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:09.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:09.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:09.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:09.627+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:10.147 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:09.826+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.827213+0000) 2024-02-19T20:20:10.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:09.917+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.918631+0000) 2024-02-19T20:20:10.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:10.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:10.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:10.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:10.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:10.182+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:10.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:10.182+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:10.418+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.419724+0000) 2024-02-19T20:20:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:10.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:10.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:10.591+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:10.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:10.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:10.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:10.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:11.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:10.825+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.827394+0000) 2024-02-19T20:20:11.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:10.917+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.918783+0000) 2024-02-19T20:20:11.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:11.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:11.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:11.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:11.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:11.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:11.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:11.134+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:11.422+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.423214+0000) 2024-02-19T20:20:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:11.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:11.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:11.579+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:11.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:11.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:11.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:11.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:11.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:11.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:12.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:11.826+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.827533+0000) 2024-02-19T20:20:12.114 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:11.918+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.918955+0000) 2024-02-19T20:20:12.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:12.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:12.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:12.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:12.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:12.113+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:12.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:12.113+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:12.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:12.422+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.423497+0000) 2024-02-19T20:20:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:12.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:12.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:12.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:12.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:12.455+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:12.455+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:12.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:12.455+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:13.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:12.826+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.827717+0000) 2024-02-19T20:20:13.120 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:12.918+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.919157+0000) 2024-02-19T20:20:13.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:13.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:13.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:13.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:13.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:13.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:13.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:13.118+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:13.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:13.407+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57103 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:13.404+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.405648+0000) 2024-02-19T20:20:13.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:13.422+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.423778+0000) 2024-02-19T20:20:13.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:13.401+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.403397+0000) 2024-02-19T20:20:13.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:13.547+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:13.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:13.547+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:13.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:13.547+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:13.401+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.402899+0000) 2024-02-19T20:20:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:13.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:13.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:13.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:13.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:13.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:14.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:13.826+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.827846+0000) 2024-02-19T20:20:14.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:13.917+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.919293+0000) 2024-02-19T20:20:14.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:14.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:14.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:14.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:14.401 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:14.154+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:14.402 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:14.154+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:14.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:14.424+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.425998+0000) 2024-02-19T20:20:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:14.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:14.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:14.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:14.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:14.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:14.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:14.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:14.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:15.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:14.827+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.828006+0000) 2024-02-19T20:20:15.148 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:14.918+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.919429+0000) 2024-02-19T20:20:15.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:15.167+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:15.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:15.167+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:15.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:15.167+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:15.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:15.167+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:15.427+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.428642+0000) 2024-02-19T20:20:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:15.523+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:15.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:15.523+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:15.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:15.523+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:15.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:15.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:15.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:15.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:16.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:15.826+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.828170+0000) 2024-02-19T20:20:16.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:15.918+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.919651+0000) 2024-02-19T20:20:16.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:16.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:16.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:16.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:16.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:16.178+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:16.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:16.178+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:16.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:16.430+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.431132+0000) 2024-02-19T20:20:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:16.595+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:16.595+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:16.595+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:16.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:16.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:16.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:16.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:16.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:16.512+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:17.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:16.826+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.828334+0000) 2024-02-19T20:20:17.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:16.918+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.919800+0000) 2024-02-19T20:20:17.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:17.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:17.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:17.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:17.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:17.134+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:17.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:17.134+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:17.430+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.431464+0000) 2024-02-19T20:20:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:17.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:17.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:17.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:17.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:17.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:17.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:17.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:18.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:17.827+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.828511+0000) 2024-02-19T20:20:18.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:17.919+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.919996+0000) 2024-02-19T20:20:18.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:18.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:18.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:18.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:18.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:18.184+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:18.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:18.184+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:18.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:18.416+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57175 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:18.413+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.414583+0000) 2024-02-19T20:20:18.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:18.430+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.431693+0000) 2024-02-19T20:20:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:18.409+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.410460+0000) 2024-02-19T20:20:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:18.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:18.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:18.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:18.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:18.408+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.410279+0000) 2024-02-19T20:20:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:18.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:18.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:18.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:18.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:18.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:19.149 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:18.827+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.828681+0000) 2024-02-19T20:20:19.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:18.919+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.920177+0000) 2024-02-19T20:20:19.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:19.226+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:19.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:19.226+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:19.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:19.226+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:19.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:19.226+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:19.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:19.433+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.434478+0000) 2024-02-19T20:20:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:19.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:19.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:19.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:19.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:19.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:19.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:19.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:19.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:19.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:20.149 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:19.827+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.828825+0000) 2024-02-19T20:20:20.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:19.919+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.920283+0000) 2024-02-19T20:20:20.405 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:20.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:20.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:20.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:20.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:20.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:20.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:20.209+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:20.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:20.436+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.437342+0000) 2024-02-19T20:20:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:20.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:20.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:20.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:20.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:20.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:20.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:20.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:20.584+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:21.149 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:20.919+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.920433+0000) 2024-02-19T20:20:21.149 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:20.828+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.829028+0000) 2024-02-19T20:20:21.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:21.197+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:21.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:21.197+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:21.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:21.197+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:21.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:21.197+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:21.438+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.439864+0000) 2024-02-19T20:20:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:21.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:21.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:21.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:21.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:21.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:21.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:21.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:21.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:21.583+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:22.149 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:21.828+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.829192+0000) 2024-02-19T20:20:22.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:21.919+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.920621+0000) 2024-02-19T20:20:22.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:22.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:22.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:22.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:22.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:22.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:22.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:22.194+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:22.438+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.440078+0000) 2024-02-19T20:20:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:22.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:22.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:22.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:22.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:22.428+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:22.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:22.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:22.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:22.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:23.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:22.828+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.829395+0000) 2024-02-19T20:20:23.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:22.920+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.920766+0000) 2024-02-19T20:20:23.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:23.198+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:23.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:23.198+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:23.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:23.198+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:23.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:23.199+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:23.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:23.425+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57249 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:23.422+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.423798+0000) 2024-02-19T20:20:23.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:23.438+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.440370+0000) 2024-02-19T20:20:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:23.417+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.419380+0000) 2024-02-19T20:20:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:23.575+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:23.575+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:23.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:23.575+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:23.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:23.418+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.419477+0000) 2024-02-19T20:20:23.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:23.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:23.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:23.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:23.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:23.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:24.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:23.920+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.920930+0000) 2024-02-19T20:20:24.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:23.828+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.829567+0000) 2024-02-19T20:20:24.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:24.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:24.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:24.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:24.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:24.227+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:24.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:24.227+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:24.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:24.441+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.442697+0000) 2024-02-19T20:20:24.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:24.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:24.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:24.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:24.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:24.485+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:24.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:24.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:24.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:24.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:25.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:24.828+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.829672+0000) 2024-02-19T20:20:25.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:24.919+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.921103+0000) 2024-02-19T20:20:25.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:25.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:25.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:25.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:25.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:25.254+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:25.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:25.254+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:25.443+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.445315+0000) 2024-02-19T20:20:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:25.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:25.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:25.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:25.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:25.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:25.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:25.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:25.644+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:26.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:25.829+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.829850+0000) 2024-02-19T20:20:26.150 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:25.920+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.921264+0000) 2024-02-19T20:20:26.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:26.275+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:26.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:26.275+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:26.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:26.275+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:26.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:26.275+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:26.446+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.447975+0000) 2024-02-19T20:20:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:26.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:26.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:26.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:26.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:26.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:26.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:26.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:26.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:26.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:26.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:26.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:27.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:26.829+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.830007+0000) 2024-02-19T20:20:27.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:26.920+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.921488+0000) 2024-02-19T20:20:27.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:27.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:27.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:27.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:27.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:27.281+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:27.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:27.281+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:27.447+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.448294+0000) 2024-02-19T20:20:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:27.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:27.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:27.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:27.621+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:27.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:27.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:27.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:27.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:28.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:27.829+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.830212+0000) 2024-02-19T20:20:28.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:27.921+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.921670+0000) 2024-02-19T20:20:28.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:28.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:28.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:28.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:28.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:28.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:28.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:28.253+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:28.668 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:28.434+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57322 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:28.669 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:28.431+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.432985+0000) 2024-02-19T20:20:28.669 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:28.447+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.448558+0000) 2024-02-19T20:20:28.669 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:28.426+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.428020+0000) 2024-02-19T20:20:28.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:28.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:28.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:28.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:28.670 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:28.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:28.670 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:28.426+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.428117+0000) 2024-02-19T20:20:28.670 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:28.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:28.670 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:28.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:28.671 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:28.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:28.922 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:28.829+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.830371+0000) 2024-02-19T20:20:29.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:28.921+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.921842+0000) 2024-02-19T20:20:29.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:29.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:29.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:29.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:29.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:29.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:29.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:29.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:29.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:29.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:29.706 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:29.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:29.707 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:29.264+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:29.707 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:29.451+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.452142+0000) 2024-02-19T20:20:29.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:29.921+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.922003+0000) 2024-02-19T20:20:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:29.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:29.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:29.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:29.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:29.705+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:29.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:29.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.830532+0000) 2024-02-19T20:20:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:30.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:30.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:30.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:30.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:30.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:30.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:30.242+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:30.242+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:30.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:30.454+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.454985+0000) 2024-02-19T20:20:31.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:30.921+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.922205+0000) 2024-02-19T20:20:31.151 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:30.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:31.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:30.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:31.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:30.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:31.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:30.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.830700+0000) 2024-02-19T20:20:31.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:31.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:31.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:31.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:31.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:31.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:31.417 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:31.266+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:31.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:31.266+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:31.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:31.266+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:31.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:31.267+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:31.457+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.457478+0000) 2024-02-19T20:20:31.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:31.921+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.922407+0000) 2024-02-19T20:20:31.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:31.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:31.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:31.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:31.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:31.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:31.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:31.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.830841+0000) 2024-02-19T20:20:32.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:32.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:32.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:32.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:32.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:32.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:32.278+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:32.278+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:32.278+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:32.278+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:32.457+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.457683+0000) 2024-02-19T20:20:33.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:32.922+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.922612+0000) 2024-02-19T20:20:33.151 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:32.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:33.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:32.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:33.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:32.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:33.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:32.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.830941+0000) 2024-02-19T20:20:33.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:33.238+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:33.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:33.238+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:33.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:33.238+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:33.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:33.238+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:33.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:33.443+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57396 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:33.436+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.437143+0000) 2024-02-19T20:20:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:33.439+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.440364+0000) 2024-02-19T20:20:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:33.457+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.457909+0000) 2024-02-19T20:20:33.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:33.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:33.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:33.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:33.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:33.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:33.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:33.436+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.437038+0000) 2024-02-19T20:20:34.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:33.922+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.922804+0000) 2024-02-19T20:20:34.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:33.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:34.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:33.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:34.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:33.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:34.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:33.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.831069+0000) 2024-02-19T20:20:34.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:34.288+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:34.418 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:34.288+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:34.419 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:34.288+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:34.419 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:34.288+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:34.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:34.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:34.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:34.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:34.419 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:34.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:34.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:34.459+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.460199+0000) 2024-02-19T20:20:35.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:34.922+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.922958+0000) 2024-02-19T20:20:35.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:34.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:35.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:34.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:35.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:34.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:35.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:34.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.831212+0000) 2024-02-19T20:20:35.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:35.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:35.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:35.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:35.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:35.352+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:35.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:35.337+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:35.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:35.337+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:35.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:35.337+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:35.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:35.337+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:35.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:35.462+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.463012+0000) 2024-02-19T20:20:36.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:35.922+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.923174+0000) 2024-02-19T20:20:36.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:35.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:36.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:35.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:36.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:35.780+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:36.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:35.830+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.831391+0000) 2024-02-19T20:20:36.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:36.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:36.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:36.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:36.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:36.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:36.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:36.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:36.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:36.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:36.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:36.378+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:36.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:36.379+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:36.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:36.465+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.465619+0000) 2024-02-19T20:20:37.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:36.922+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.923361+0000) 2024-02-19T20:20:37.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:36.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:37.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:36.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:37.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:36.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:37.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:36.831+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.831556+0000) 2024-02-19T20:20:37.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:37.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:37.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:37.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:37.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:37.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:37.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:37.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:37.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:37.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:37.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:37.351+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:37.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:37.352+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:37.465+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.465948+0000) 2024-02-19T20:20:38.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:37.923+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.923551+0000) 2024-02-19T20:20:38.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:37.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:38.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:37.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:38.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:37.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:38.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:37.831+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.831788+0000) 2024-02-19T20:20:38.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:38.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:38.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:38.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:38.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:38.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:38.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:38.325+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:38.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:38.325+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:38.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:38.325+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:38.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:38.325+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:38.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:38.450+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57468 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:38.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:38.444+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.445292+0000) 2024-02-19T20:20:38.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:38.444+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.445003+0000) 2024-02-19T20:20:38.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:38.448+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.448524+0000) 2024-02-19T20:20:38.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:38.465+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.466170+0000) 2024-02-19T20:20:39.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:38.923+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.923666+0000) 2024-02-19T20:20:39.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:38.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:39.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:38.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:39.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:38.823+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:39.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:38.831+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.831980+0000) 2024-02-19T20:20:39.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:39.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:39.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:39.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:39.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:39.310+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:39.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:39.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:39.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:39.342+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:39.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:39.468+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.468623+0000) 2024-02-19T20:20:40.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:39.923+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.923797+0000) 2024-02-19T20:20:40.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:39.831+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.832151+0000) 2024-02-19T20:20:40.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:39.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:40.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:39.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:40.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:39.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:40.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:40.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:40.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:40.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:40.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:40.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:40.309+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:40.309+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:40.309+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:40.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:40.309+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:40.470+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.471261+0000) 2024-02-19T20:20:41.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:40.923+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.923958+0000) 2024-02-19T20:20:41.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:40.831+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.832327+0000) 2024-02-19T20:20:41.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:40.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:41.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:40.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:41.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:40.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:41.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:41.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:41.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:41.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:41.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:41.319+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:41.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:41.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:41.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:41.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:41.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:41.272+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:41.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:41.272+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:41.473+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.473674+0000) 2024-02-19T20:20:42.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:41.923+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.924112+0000) 2024-02-19T20:20:42.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:41.832+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.832519+0000) 2024-02-19T20:20:42.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:41.853+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:42.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:41.853+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:42.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:41.853+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:42.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:42.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:42.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:42.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:42.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:42.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:42.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:42.245+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:42.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:42.245+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:42.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:42.245+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:42.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:42.245+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:42.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:42.473+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.473959+0000) 2024-02-19T20:20:43.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:42.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.924270+0000) 2024-02-19T20:20:43.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:42.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:43.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:42.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:43.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:42.808+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:43.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:42.832+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.832695+0000) 2024-02-19T20:20:43.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:43.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:43.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:43.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:43.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:43.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:43.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:43.276+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:43.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:43.276+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:43.426 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:43.276+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:43.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:43.276+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:43.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:43.460+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57542 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:43.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:43.452+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.452578+0000) 2024-02-19T20:20:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:43.455+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.456265+0000) 2024-02-19T20:20:43.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:43.473+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.474174+0000) 2024-02-19T20:20:43.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:43.452+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.453290+0000) 2024-02-19T20:20:44.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:43.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.924490+0000) 2024-02-19T20:20:44.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:43.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:44.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:43.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:44.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:43.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:44.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:43.832+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.832818+0000) 2024-02-19T20:20:44.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:44.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:44.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:44.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:44.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:44.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:44.298+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:44.298+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:44.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:44.298+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:44.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:44.298+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:44.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:44.477+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.477758+0000) 2024-02-19T20:20:45.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:44.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.924623+0000) 2024-02-19T20:20:45.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:44.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:45.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:44.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:45.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:44.811+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:45.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:44.832+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.832985+0000) 2024-02-19T20:20:45.430 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:45.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:45.430 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:45.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:45.430 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:45.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:45.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:45.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:45.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:45.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:45.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:45.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:45.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:45.293+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:45.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:45.480+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.480605+0000) 2024-02-19T20:20:46.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:45.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.924795+0000) 2024-02-19T20:20:46.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:45.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:46.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:45.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:46.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:45.804+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:46.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:45.832+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.833157+0000) 2024-02-19T20:20:46.430 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:46.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:46.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:46.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:46.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:46.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:46.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:46.317+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:46.432 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:46.317+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:46.432 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:46.317+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:46.432 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:46.317+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:46.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:46.483+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.483348+0000) 2024-02-19T20:20:47.154 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:46.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.924973+0000) 2024-02-19T20:20:47.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:46.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:47.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:46.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:47.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:46.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:47.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:46.832+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.833371+0000) 2024-02-19T20:20:47.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:47.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:47.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:47.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:47.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:47.346+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:47.432 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:47.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:47.432 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:47.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:47.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:47.277+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:47.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:47.277+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:47.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:47.483+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.483619+0000) 2024-02-19T20:20:48.154 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:47.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.925185+0000) 2024-02-19T20:20:48.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:47.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:48.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:47.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:48.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:47.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:48.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:47.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.833549+0000) 2024-02-19T20:20:48.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:48.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:48.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:48.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:48.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:48.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:48.431 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:48.253+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:48.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:48.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:48.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:48.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:48.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:48.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:48.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:48.462+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.463096+0000) 2024-02-19T20:20:48.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:48.469+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57618 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:48.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:48.462+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.463019+0000) 2024-02-19T20:20:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:48.465+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.465882+0000) 2024-02-19T20:20:48.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:48.483+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.483928+0000) 2024-02-19T20:20:48.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:48.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.925369+0000) 2024-02-19T20:20:48.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:48.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:48.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:48.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:48.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:48.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:48.996 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:48.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.833729+0000) 2024-02-19T20:20:49.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:49.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:49.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:49.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:49.715 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:49.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:49.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:49.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:49.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:49.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:49.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:49.293+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:49.716 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:49.293+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:49.717 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:49.485+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.486343+0000) 2024-02-19T20:20:49.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:49.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.925538+0000) 2024-02-19T20:20:49.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:49.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:49.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:49.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:49.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:49.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:49.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:49.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.833927+0000) 2024-02-19T20:20:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:50.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:50.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:50.333+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:50.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:50.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:50.321+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:50.321+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:50.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:50.488+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.488978+0000) 2024-02-19T20:20:50.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:50.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.925741+0000) 2024-02-19T20:20:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:50.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:50.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:50.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:50.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:50.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.834112+0000) 2024-02-19T20:20:51.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:51.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:51.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:51.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:51.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:51.299+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:51.707 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:51.306+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:51.708 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:51.306+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:51.708 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:51.306+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:51.708 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:51.306+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:51.708 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:51.491+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.491615+0000) 2024-02-19T20:20:51.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:51.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.925928+0000) 2024-02-19T20:20:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:51.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:51.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:51.706+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:51.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:51.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.834266+0000) 2024-02-19T20:20:52.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:52.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:52.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:52.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:52.703 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:52.298+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:52.704 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:52.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:52.704 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:52.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:52.704 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:52.265+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:52.704 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:52.265+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:52.705 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:52.490+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.491815+0000) 2024-02-19T20:20:52.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:52.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.926114+0000) 2024-02-19T20:20:52.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:52.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:52.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:52.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:52.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:52.701+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:52.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:52.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.834440+0000) 2024-02-19T20:20:53.597 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:53.475+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57692 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:53.598 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:53.469+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.471184+0000) 2024-02-19T20:20:53.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:53.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:53.598 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:53.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:53.599 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:53.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:53.599 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:53.469+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.471041+0000) 2024-02-19T20:20:53.599 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:53.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:53.599 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:53.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:53.599 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:53.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:53.599 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:53.253+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:53.600 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:53.473+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.475456+0000) 2024-02-19T20:20:53.600 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:53.490+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.492006+0000) 2024-02-19T20:20:53.926 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:53.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:53.926 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:53.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:53.926 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:53.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:53.927 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:53.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.834604+0000) 2024-02-19T20:20:54.241 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:53.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.926235+0000) 2024-02-19T20:20:54.496 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:54.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:54.496 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:54.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:54.496 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:54.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:54.496 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:54.267+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:54.496 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:54.267+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:54.497 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:54.267+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:54.497 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:54.267+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:54.834 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:54.493+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.494542+0000) 2024-02-19T20:20:54.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:54.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:54.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:54.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:54.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:54.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:55.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:54.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.834746+0000) 2024-02-19T20:20:55.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:54.924+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.926389+0000) 2024-02-19T20:20:55.438 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:55.244+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:55.438 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:55.244+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:55.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:55.244+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:55.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:55.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:55.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:55.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:55.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:55.251+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:55.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:55.251+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:55.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:55.496+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.497731+0000) 2024-02-19T20:20:56.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:55.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.926629+0000) 2024-02-19T20:20:56.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:55.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:56.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:55.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:56.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:55.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:56.157 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:55.833+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.834921+0000) 2024-02-19T20:20:56.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:56.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:56.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:56.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:56.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:56.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:56.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:56.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:56.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:56.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:56.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:56.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:56.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:56.221+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:56.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:56.498+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.500293+0000) 2024-02-19T20:20:57.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:56.925+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.926798+0000) 2024-02-19T20:20:57.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:56.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:57.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:56.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:57.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:56.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:57.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:56.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.835126+0000) 2024-02-19T20:20:57.438 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:57.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:57.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:57.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:57.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:57.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:57.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:57.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:57.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:57.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:57.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:57.206+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:57.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:57.206+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:57.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:57.499+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.500604+0000) 2024-02-19T20:20:58.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:57.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.926988+0000) 2024-02-19T20:20:58.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:57.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:58.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:57.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:58.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:57.782+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:58.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:57.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.835311+0000) 2024-02-19T20:20:58.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:58.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:58.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:58.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:58.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:58.176+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:58.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:58.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:58.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:58.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:58.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:58.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:58.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:58.183+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:58.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:20:58.483+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57765 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:20:58.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:58.477+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.478835+0000) 2024-02-19T20:20:58.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:58.477+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.478608+0000) 2024-02-19T20:20:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:58.480+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.481950+0000) 2024-02-19T20:20:58.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:58.499+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.500849+0000) 2024-02-19T20:20:59.156 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:58.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.927192+0000) 2024-02-19T20:20:59.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:58.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:59.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:58.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:59.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:58.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:20:59.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:58.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.835466+0000) 2024-02-19T20:20:59.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:59.164+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:59.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:59.164+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:59.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:59.164+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:20:59.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:59.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:59.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:59.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:59.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:59.214+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:20:59.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:59.214+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:59.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:20:59.501+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.503168+0000) 2024-02-19T20:21:00.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:20:59.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.927327+0000) 2024-02-19T20:21:00.124 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:00.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:00.124 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:59.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:00.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:59.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:00.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:59.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:00.125 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:20:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:20:59.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.835626+0000) 2024-02-19T20:21:00.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:00.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:00.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:00.122+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:00.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:00.241+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:00.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:00.241+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:00.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:00.241+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:00.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:00.241+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:00.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:00.504+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.505863+0000) 2024-02-19T20:21:01.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:00.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.927500+0000) 2024-02-19T20:21:01.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:01.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:01.108 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:00.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.835784+0000) 2024-02-19T20:21:01.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:00.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:01.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:00.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:01.109 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:00.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:01.442 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:01.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:01.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:01.107+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:01.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:01.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:01.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:01.235+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:01.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:01.236+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:01.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:01.236+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:01.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:01.506+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.508416+0000) 2024-02-19T20:21:02.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:01.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.927704+0000) 2024-02-19T20:21:02.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:02.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:02.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:01.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:02.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:01.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:02.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:01.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:02.130 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:01.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.835902+0000) 2024-02-19T20:21:02.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:02.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:02.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:02.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:02.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:02.285+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:02.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:02.285+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:02.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:02.285+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:02.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:02.285+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:02.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:02.507+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.508721+0000) 2024-02-19T20:21:03.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:02.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.927851+0000) 2024-02-19T20:21:03.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:03.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:03.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:02.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.836055+0000) 2024-02-19T20:21:03.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:02.857+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:03.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:02.857+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:03.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:02.857+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:03.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:03.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:03.443 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:03.136+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:03.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:03.314+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:03.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:03.314+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:03.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:03.314+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:03.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:03.314+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:03.709 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:03.492+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57839 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:03.710 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:03.485+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.487045+0000) 2024-02-19T20:21:03.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:03.485+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.487157+0000) 2024-02-19T20:21:03.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:03.488+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.490409+0000) 2024-02-19T20:21:03.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:03.507+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.508919+0000) 2024-02-19T20:21:03.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:03.927+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.928048+0000) 2024-02-19T20:21:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:03.834+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.836270+0000) 2024-02-19T20:21:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:03.892+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:03.892+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:03.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:03.892+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:04.336 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:04.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:04.336 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:04.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:04.336 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:04.171+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:04.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:04.334+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:04.334+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:04.334+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:04.334+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:04.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:04.510+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.511363+0000) 2024-02-19T20:21:05.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:04.835+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.836366+0000) 2024-02-19T20:21:05.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:04.908+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:05.133 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:04.908+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:05.134 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:04.908+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:05.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:04.926+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.928205+0000) 2024-02-19T20:21:05.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:05.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:05.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:05.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:05.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:05.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:05.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:05.376+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:05.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:05.376+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:05.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:05.376+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:05.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:05.377+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:05.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:05.513+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.514323+0000) 2024-02-19T20:21:06.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:05.927+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.928404+0000) 2024-02-19T20:21:06.157 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:05.835+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.836526+0000) 2024-02-19T20:21:06.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:05.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:06.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:05.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:06.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:05.873+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:06.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:06.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:06.447 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:06.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:06.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:06.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:06.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:06.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:06.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:06.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:06.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:06.386+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:06.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:06.386+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:06.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:06.516+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.516863+0000) 2024-02-19T20:21:07.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:06.927+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.928566+0000) 2024-02-19T20:21:07.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:06.834+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:07.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:06.834+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:07.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:06.834+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:07.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:06.835+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.836623+0000) 2024-02-19T20:21:07.426 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:07.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:07.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:07.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:07.427 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:07.215+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:07.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:07.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:07.425+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:07.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:07.425+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:07.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:07.516+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.517120+0000) 2024-02-19T20:21:08.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:07.928+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.928769+0000) 2024-02-19T20:21:08.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:07.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.836789+0000) 2024-02-19T20:21:08.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:07.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:08.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:07.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:08.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:07.848+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:08.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:08.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:08.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:08.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:08.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:08.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:08.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:08.401+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:08.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:08.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:08.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:08.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:08.448 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:08.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:08.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:08.500+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57913 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:08.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:08.494+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.495503+0000) 2024-02-19T20:21:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:08.498+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.500078+0000) 2024-02-19T20:21:08.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:08.515+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.517377+0000) 2024-02-19T20:21:08.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:08.494+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.495939+0000) 2024-02-19T20:21:09.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:08.927+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.928971+0000) 2024-02-19T20:21:09.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:08.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:09.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:08.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:09.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:08.819+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:09.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:08.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.836960+0000) 2024-02-19T20:21:09.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:09.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:09.435 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:09.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:09.436 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:09.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:09.434+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:09.434+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:09.434+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:09.434+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:09.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:09.518+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.519883+0000) 2024-02-19T20:21:10.151 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:09.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:10.151 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:09.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:10.151 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:09.824+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:10.151 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:09.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.837079+0000) 2024-02-19T20:21:10.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:09.928+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.929066+0000) 2024-02-19T20:21:10.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:10.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:10.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:10.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:10.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:10.149+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:10.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:10.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:10.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:10.449+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:10.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:10.521+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.522651+0000) 2024-02-19T20:21:11.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:10.927+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.929254+0000) 2024-02-19T20:21:11.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:11.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:11.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:10.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:11.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:10.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:11.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:10.795+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:11.136 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:10.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.837257+0000) 2024-02-19T20:21:11.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:11.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:11.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:11.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:11.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:11.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:11.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:11.444+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:11.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:11.524+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.525281+0000) 2024-02-19T20:21:12.004 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:11.928+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.929374+0000) 2024-02-19T20:21:12.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:11.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:12.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:11.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:12.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:11.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:12.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:11.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.837494+0000) 2024-02-19T20:21:12.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:12.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:12.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:12.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:12.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:12.120+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:12.712 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:12.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:12.712 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:12.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:12.712 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:12.476+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:12.712 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:12.476+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:12.712 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:12.524+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.525606+0000) 2024-02-19T20:21:12.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:12.928+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.929572+0000) 2024-02-19T20:21:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:12.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:12.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:12.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:12.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:12.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:12.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.837643+0000) 2024-02-19T20:21:13.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:13.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:13.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:13.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:13.452 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:13.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:13.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:13.501+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.503338+0000) 2024-02-19T20:21:13.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:13.509+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 57986 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:13.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:13.502+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.503701+0000) 2024-02-19T20:21:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:13.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:13.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:13.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:13.478+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:13.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:13.506+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.507611+0000) 2024-02-19T20:21:13.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:13.524+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.525918+0000) 2024-02-19T20:21:14.068 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:13.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:14.069 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:13.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:14.069 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:13.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:14.069 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:13.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.837747+0000) 2024-02-19T20:21:14.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:13.929+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.929760+0000) 2024-02-19T20:21:14.070 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:14.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:14.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:14.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:14.071 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:14.066+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:14.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:14.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:14.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:14.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:14.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:14.504+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:14.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:14.504+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:14.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:14.526+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.528335+0000) 2024-02-19T20:21:14.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:14.929+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.929927+0000) 2024-02-19T20:21:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:14.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:14.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:14.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:14.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:14.836+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.837898+0000) 2024-02-19T20:21:15.455 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:15.052+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:15.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:15.052+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:15.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:15.052+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:15.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:15.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:15.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:15.698+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:15.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:15.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:15.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:15.467+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:15.467+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:15.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:15.530+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.531168+0000) 2024-02-19T20:21:16.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:15.837+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.838113+0000) 2024-02-19T20:21:16.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:15.929+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.930108+0000) 2024-02-19T20:21:16.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:16.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:16.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:16.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:16.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:16.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:16.652+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:16.445+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:16.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:16.445+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:16.445+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:16.445+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:16.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:16.532+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.533959+0000) 2024-02-19T20:21:17.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:16.837+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.838276+0000) 2024-02-19T20:21:17.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.929+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.930314+0000) 2024-02-19T20:21:17.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:17.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:17.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:16.992+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:17.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:17.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:17.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:17.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:17.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:17.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:17.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:17.444+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:17.444+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:17.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:17.533+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.534296+0000) 2024-02-19T20:21:18.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:17.837+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.838479+0000) 2024-02-19T20:21:18.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:17.929+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.930514+0000) 2024-02-19T20:21:18.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:17.988+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:18.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:17.988+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:18.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:17.988+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:18.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:18.518+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58060 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:18.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:18.511+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.513393+0000) 2024-02-19T20:21:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:18.511+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.512875+0000) 2024-02-19T20:21:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:18.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:18.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:18.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:18.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:18.437+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:18.437+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:18.437+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:18.437+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:18.730 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:18.515+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.516696+0000) 2024-02-19T20:21:18.731 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:18.533+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.534527+0000) 2024-02-19T20:21:19.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:18.837+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.838607+0000) 2024-02-19T20:21:19.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:18.930+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.930718+0000) 2024-02-19T20:21:19.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:19.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:19.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:19.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:19.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:19.018+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:19.639+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:19.639+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:19.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:19.639+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:19.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:19.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:19.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:19.478+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:19.478+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:19.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:19.537+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.538198+0000) 2024-02-19T20:21:20.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:19.838+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.838775+0000) 2024-02-19T20:21:20.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:19.930+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.930873+0000) 2024-02-19T20:21:20.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:20.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:20.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:20.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:20.157 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:20.022+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:20.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:20.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:20.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:20.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:20.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:20.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:20.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:20.482+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:20.482+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:20.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:20.540+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.540924+0000) 2024-02-19T20:21:21.159 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:20.838+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.838945+0000) 2024-02-19T20:21:21.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:20.930+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.931001+0000) 2024-02-19T20:21:21.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:21.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:21.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:21.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:21.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:21.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:21.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:21.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:21.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:21.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:21.526+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:21.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:21.526+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:21.526+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:21.526+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:21.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:21.543+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.543622+0000) 2024-02-19T20:21:22.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:21.838+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.839170+0000) 2024-02-19T20:21:22.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:21.930+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.931156+0000) 2024-02-19T20:21:22.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:22.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:22.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:22.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:22.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:22.059+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:22.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:22.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:22.687+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:22.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:22.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:22.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:22.534+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:22.534+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:22.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:22.543+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.543924+0000) 2024-02-19T20:21:23.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:22.838+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.839365+0000) 2024-02-19T20:21:23.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:22.930+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.931322+0000) 2024-02-19T20:21:23.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:23.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:23.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:23.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:23.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:23.043+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:23.711 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:23.527+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58133 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:23.712 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:23.521+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.522274+0000) 2024-02-19T20:21:23.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:23.520+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.521155+0000) 2024-02-19T20:21:23.712 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:23.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:23.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:23.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:23.524+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.525237+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:23.543+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.544210+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:23.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:23.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:23.713 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:23.558+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:23.714 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:23.558+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:23.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:23.839+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.839542+0000) 2024-02-19T20:21:23.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:23.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.931454+0000) 2024-02-19T20:21:24.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:24.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:24.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:24.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:24.461 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:24.024+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:24.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:24.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:24.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:24.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:24.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:24.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:24.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:24.542+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:24.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:24.546+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.546512+0000) 2024-02-19T20:21:24.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:24.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.931562+0000) 2024-02-19T20:21:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:24.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:24.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:24.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:24.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:24.839+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.839685+0000) 2024-02-19T20:21:25.464 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:25.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:25.464 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:25.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:25.464 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:25.068+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:25.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:25.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:25.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:25.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:25.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:25.548+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.549133+0000) 2024-02-19T20:21:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:25.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:25.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:25.573+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:25.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:25.573+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:26.161 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:25.839+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.839845+0000) 2024-02-19T20:21:26.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:25.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.931722+0000) 2024-02-19T20:21:26.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:26.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:26.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:26.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:26.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:26.070+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:26.551+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.551767+0000) 2024-02-19T20:21:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:26.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:26.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:26.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:26.598+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:26.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:26.598+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:27.113 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:26.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.931903+0000) 2024-02-19T20:21:27.113 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:27.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:27.113 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:26.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:27.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:26.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:27.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:26.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:27.114 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:26.839+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.840073+0000) 2024-02-19T20:21:27.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:27.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:27.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:27.112+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:27.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:27.551+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.552060+0000) 2024-02-19T20:21:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:27.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:27.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:27.579+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:27.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:27.579+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:28.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:27.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.932105+0000) 2024-02-19T20:21:28.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:28.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:28.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:27.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:28.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:27.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:28.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:27.781+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:28.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:27.839+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.840278+0000) 2024-02-19T20:21:28.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:28.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:28.465 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:28.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:28.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:28.528+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.529179+0000) 2024-02-19T20:21:28.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:28.529+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.529956+0000) 2024-02-19T20:21:28.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:28.535+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58207 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:28.532+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.532977+0000) 2024-02-19T20:21:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:28.551+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.552296+0000) 2024-02-19T20:21:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:28.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:28.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:28.599+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:28.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:28.600+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:29.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:28.789+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:29.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:28.789+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:29.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:28.789+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:29.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:28.839+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.840421+0000) 2024-02-19T20:21:29.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:28.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.932263+0000) 2024-02-19T20:21:29.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:29.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:29.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:29.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:29.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:29.127+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:29.554+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.554656+0000) 2024-02-19T20:21:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:29.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:29.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:29.587+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:29.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:29.587+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:30.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:29.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:30.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:29.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:30.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:29.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:30.152 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:29.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.840545+0000) 2024-02-19T20:21:30.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:29.931+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.932420+0000) 2024-02-19T20:21:30.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:30.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:30.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:30.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:30.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:30.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:30.557+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.557668+0000) 2024-02-19T20:21:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:30.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:30.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:30.632+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:30.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:30.632+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:31.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:30.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:31.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:30.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:31.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:30.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:31.142 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:30.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.840703+0000) 2024-02-19T20:21:31.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:30.932+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.932618+0000) 2024-02-19T20:21:31.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:31.141+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:31.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:31.141+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:31.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:31.141+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:31.561+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.561508+0000) 2024-02-19T20:21:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:31.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:31.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:31.606+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:31.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:31.606+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:32.162 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:31.932+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.932769+0000) 2024-02-19T20:21:32.163 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:31.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:32.163 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:31.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:32.163 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:31.800+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:32.163 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:31.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.840849+0000) 2024-02-19T20:21:32.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:32.170+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:32.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:32.170+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:32.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:32.170+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:32.561+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.561718+0000) 2024-02-19T20:21:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:32.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:32.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:32.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:32.561+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:32.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:32.561+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:33.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:32.932+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.932948+0000) 2024-02-19T20:21:33.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:33.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:33.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:32.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:33.161 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:32.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:33.161 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:32.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:33.161 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:32.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.840998+0000) 2024-02-19T20:21:33.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:33.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:33.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:33.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:33.728 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:33.543+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58280 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:33.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:33.537+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.537491+0000) 2024-02-19T20:21:33.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:33.537+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.537849+0000) 2024-02-19T20:21:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:33.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:33.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:33.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:33.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:33.542+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.542708+0000) 2024-02-19T20:21:33.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:33.561+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.561919+0000) 2024-02-19T20:21:34.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:33.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:34.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:33.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:34.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:33.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:34.118 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:33.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.841110+0000) 2024-02-19T20:21:34.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:33.932+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.933125+0000) 2024-02-19T20:21:34.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:34.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:34.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:34.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:34.119 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:34.117+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:34.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:34.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:34.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:34.500+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:34.563+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.564186+0000) 2024-02-19T20:21:35.143 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:34.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:35.143 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:34.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:35.143 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:34.783+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:35.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:34.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.841198+0000) 2024-02-19T20:21:35.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:34.932+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.933277+0000) 2024-02-19T20:21:35.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:35.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:35.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:35.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:35.144 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:35.142+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:35.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:35.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:35.542+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:35.542+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:35.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:35.567+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.567936+0000) 2024-02-19T20:21:36.002 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:35.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.933472+0000) 2024-02-19T20:21:36.003 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:35.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:36.003 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:35.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:36.003 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:35.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:36.003 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:35.840+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.841326+0000) 2024-02-19T20:21:36.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:36.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:36.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:36.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:36.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:36.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:36.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:36.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:36.566+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:36.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:36.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:36.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:36.570+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.570722+0000) 2024-02-19T20:21:37.134 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:36.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.933641+0000) 2024-02-19T20:21:37.135 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:37.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:37.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:36.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:37.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:36.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:37.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:36.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:37.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:36.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.841456+0000) 2024-02-19T20:21:37.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:37.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:37.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:37.134+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:37.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:37.570+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.571044+0000) 2024-02-19T20:21:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:37.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:37.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:37.615+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:37.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:37.615+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:38.155 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:37.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.933782+0000) 2024-02-19T20:21:38.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:37.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:38.155 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:37.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:38.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:37.761+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:38.156 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:37.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.841579+0000) 2024-02-19T20:21:38.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:38.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:38.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:38.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:38.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:38.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:38.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:38.553+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58354 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:38.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:38.545+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.546217+0000) 2024-02-19T20:21:38.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:38.546+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.546557+0000) 2024-02-19T20:21:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:38.549+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.549745+0000) 2024-02-19T20:21:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:38.570+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.571195+0000) 2024-02-19T20:21:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:38.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:38.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:38.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:38.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:38.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:39.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:38.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.933969+0000) 2024-02-19T20:21:39.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:38.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:39.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:38.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:39.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:38.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:39.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:38.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.841789+0000) 2024-02-19T20:21:39.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:39.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:39.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:39.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:39.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:39.167+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:39.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:39.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:39.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:39.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:39.574+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.574831+0000) 2024-02-19T20:21:39.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:39.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:39.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:39.642+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:39.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:39.642+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:40.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:39.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.841886+0000) 2024-02-19T20:21:40.158 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:39.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.934085+0000) 2024-02-19T20:21:40.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:40.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:40.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:40.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:40.184+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:40.578+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.578552+0000) 2024-02-19T20:21:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:40.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:40.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:40.649+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:40.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:40.649+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:41.003 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:40.934+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.934291+0000) 2024-02-19T20:21:41.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:40.752+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:41.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:40.752+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:41.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:40.752+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:41.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:40.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.842006+0000) 2024-02-19T20:21:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:41.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:41.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:41.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:41.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:41.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:41.581+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.581457+0000) 2024-02-19T20:21:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:41.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:41.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:41.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:41.644+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:41.729 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:41.644+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:42.004 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:41.934+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.934485+0000) 2024-02-19T20:21:42.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:41.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:42.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:41.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:42.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:41.732+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:42.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:41.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.842209+0000) 2024-02-19T20:21:42.388 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:42.241+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:42.388 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:42.241+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:42.388 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:42.241+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:42.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:42.581+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.581774+0000) 2024-02-19T20:21:42.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:42.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:42.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:42.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:42.684+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:42.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:42.684+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:42.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:42.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.934668+0000) 2024-02-19T20:21:42.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:42.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:42.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:42.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:42.979 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:42.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.842338+0000) 2024-02-19T20:21:43.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:43.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:43.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:43.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:43.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:43.750 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:43.560+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58427 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:43.751 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:43.553+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.555077+0000) 2024-02-19T20:21:43.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:43.554+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.555989+0000) 2024-02-19T20:21:43.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:43.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:43.751 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:43.557+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.559272+0000) 2024-02-19T20:21:43.751 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:43.580+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.582028+0000) 2024-02-19T20:21:43.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:43.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:43.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:43.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:43.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:43.673+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:43.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:43.673+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:44.004 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:43.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.934820+0000) 2024-02-19T20:21:44.004 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:43.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:44.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:43.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:44.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:43.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.842526+0000) 2024-02-19T20:21:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:44.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:44.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:44.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:44.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:44.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:44.583+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.584344+0000) 2024-02-19T20:21:44.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:44.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:44.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:44.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:44.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:44.686+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:44.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:44.686+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:45.164 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:44.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.934990+0000) 2024-02-19T20:21:45.164 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:44.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:45.164 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:44.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:45.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:44.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:45.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:44.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.842622+0000) 2024-02-19T20:21:45.460 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:45.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:45.460 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:45.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:45.460 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:45.193+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:45.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:45.586+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.586998+0000) 2024-02-19T20:21:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:45.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:45.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:45.682+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:45.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:45.682+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:46.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:45.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:46.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:45.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:46.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:45.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:46.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:45.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.842843+0000) 2024-02-19T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:45.933+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.935180+0000) 2024-02-19T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:46.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:46.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:46.161 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:46.158+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:46.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:46.589+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.590671+0000) 2024-02-19T20:21:46.774 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:46.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:46.774 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:46.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:46.774 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:46.696+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:46.774 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:46.696+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:47.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:46.934+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.935387+0000) 2024-02-19T20:21:47.153 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:47.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:47.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:46.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:47.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:46.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:47.153 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:46.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:47.154 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:46.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.842956+0000) 2024-02-19T20:21:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:47.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:47.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:47.151+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:47.754 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:47.590+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.590970+0000) 2024-02-19T20:21:47.755 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:47.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:47.755 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:47.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:47.755 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:47.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:47.755 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:47.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:48.005 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:47.934+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.935609+0000) 2024-02-19T20:21:48.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:47.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:48.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:47.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:48.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:47.753+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:48.006 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:47.842+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.843135+0000) 2024-02-19T20:21:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:48.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:48.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:48.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:48.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:48.843 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:48.569+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58501 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:48.844 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:48.562+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.563676+0000) 2024-02-19T20:21:48.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:48.562+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.564125+0000) 2024-02-19T20:21:48.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:48.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:48.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:48.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:48.722+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:48.565+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.567045+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:48.589+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.591277+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:48.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:48.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:48.618+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:48.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:48.618+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:49.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:48.841+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.843287+0000) 2024-02-19T20:21:49.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:48.934+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.935804+0000) 2024-02-19T20:21:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:49.209+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:49.209+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:49.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:49.209+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:49.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:49.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:49.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:49.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:49.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:49.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:49.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:49.593+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.594887+0000) 2024-02-19T20:21:49.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:49.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:49.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:49.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:49.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:49.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:49.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:49.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:50.158 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:49.842+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.843494+0000) 2024-02-19T20:21:50.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:49.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.935944+0000) 2024-02-19T20:21:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:50.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:50.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:50.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:50.237+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:50.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:50.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:50.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:50.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:50.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:50.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:50.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:50.596+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.597649+0000) 2024-02-19T20:21:50.844 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:50.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:50.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:50.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:50.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:50.653+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:50.845 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:50.653+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:51.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:50.842+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.843648+0000) 2024-02-19T20:21:51.169 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:50.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.936116+0000) 2024-02-19T20:21:51.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:51.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:51.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:51.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:51.254+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:51.760 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:51.599+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.600433+0000) 2024-02-19T20:21:51.760 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:51.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:51.761 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:51.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:51.761 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:51.668+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:51.761 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:51.668+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:52.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:51.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.936285+0000) 2024-02-19T20:21:52.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:51.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:52.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:51.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:52.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:51.759+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:52.165 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:51.842+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.843817+0000) 2024-02-19T20:21:52.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:52.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:52.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:52.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:52.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:52.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:52.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:52.599+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.600790+0000) 2024-02-19T20:21:52.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:52.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:52.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:52.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:52.780 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:52.621+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:52.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:52.621+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:53.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:52.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.936493+0000) 2024-02-19T20:21:53.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:52.779+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:53.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:52.779+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:53.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:52.779+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:53.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:52.843+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.844029+0000) 2024-02-19T20:21:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:53.249+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:53.249+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:53.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:53.249+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:53.751 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:53.578+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58574 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:53.751 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:53.571+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.573297+0000) 2024-02-19T20:21:53.751 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:53.571+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.573364+0000) 2024-02-19T20:21:53.752 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:53.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:53.752 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:53.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:53.752 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:53.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:53.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:53.575+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.576572+0000) 2024-02-19T20:21:53.752 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:53.599+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.601147+0000) 2024-02-19T20:21:53.753 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:53.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:53.753 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:53.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:53.753 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:53.662+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:53.753 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:53.662+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:54.006 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:53.842+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.844218+0000) 2024-02-19T20:21:54.006 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:53.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.936696+0000) 2024-02-19T20:21:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:54.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:54.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:54.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:54.224+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:54.772 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:54.602+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.603464+0000) 2024-02-19T20:21:54.772 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:54.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:54.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:54.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:54.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:54.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:54.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:54.666+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:55.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:54.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.936828+0000) 2024-02-19T20:21:55.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:54.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:55.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:54.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:55.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:54.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:55.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:54.842+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.844384+0000) 2024-02-19T20:21:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:55.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:55.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:55.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:55.188+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:55.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:55.605+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.606087+0000) 2024-02-19T20:21:55.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:55.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:55.773 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:55.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:55.774 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:55.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:55.774 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:55.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:56.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:55.936+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.937025+0000) 2024-02-19T20:21:56.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:55.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:56.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:55.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:56.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:55.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:56.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:55.843+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.844574+0000) 2024-02-19T20:21:56.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:56.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:56.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:56.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:56.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:56.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:56.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:56.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:56.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:56.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:56.812 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:56.736+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:56.607+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.608949+0000) 2024-02-19T20:21:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:56.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:56.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:56.723+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:56.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:56.723+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:57.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:56.843+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.844698+0000) 2024-02-19T20:21:57.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:56.935+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.937197+0000) 2024-02-19T20:21:57.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:57.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:57.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:57.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:57.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:57.747 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:57.608+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.609316+0000) 2024-02-19T20:21:58.007 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:57.936+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.937318+0000) 2024-02-19T20:21:58.007 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:57.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:58.007 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:57.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:58.008 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:57.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:58.008 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:57.844+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.844865+0000) 2024-02-19T20:21:58.008 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:57.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:58.008 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:57.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:58.008 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:57.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:58.008 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:57.750+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:58.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:58.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:58.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:58.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:58.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:58.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:58.582 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:58.580+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.581771+0000) 2024-02-19T20:21:58.845 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:21:58.586+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58648 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:21:58.845 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:58.581+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.582674+0000) 2024-02-19T20:21:58.845 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:58.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:58.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:58.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:58.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:58.769+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:21:58.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:58.583+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.584837+0000) 2024-02-19T20:21:58.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:58.608+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.609570+0000) 2024-02-19T20:21:58.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:58.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:58.846 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:58.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:58.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:58.749+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:21:58.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:58.749+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:59.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:58.843+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.845039+0000) 2024-02-19T20:21:59.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:58.936+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.937484+0000) 2024-02-19T20:21:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:59.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:59.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:59.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:59.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:21:59.775 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:59.611+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.611961+0000) 2024-02-19T20:22:00.159 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:21:59.936+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.937594+0000) 2024-02-19T20:22:00.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:59.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:00.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:59.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:00.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:59.799+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:00.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:21:59.844+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.845185+0000) 2024-02-19T20:22:00.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:59.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:00.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:59.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:00.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:59.774+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:00.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:21:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:21:59.774+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:00.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:00.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:00.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:00.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:00.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:00.202+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:00.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:00.613+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.614623+0000) 2024-02-19T20:22:00.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:00.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:00.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:00.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:00.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:00.742+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:00.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:00.742+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:01.166 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:00.937+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.937773+0000) 2024-02-19T20:22:01.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:00.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:01.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:00.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:01.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:00.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:01.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:00.843+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.845378+0000) 2024-02-19T20:22:01.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:01.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:01.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:01.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:01.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:01.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:01.797 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:01.617+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.618293+0000) 2024-02-19T20:22:01.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:01.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:01.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:01.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:01.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:01.727+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:01.798 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:01.727+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:02.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:01.936+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.937914+0000) 2024-02-19T20:22:02.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:01.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:02.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:01.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:02.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:01.796+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:02.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:01.844+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.845596+0000) 2024-02-19T20:22:02.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:02.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:02.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:02.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:02.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:02.617+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.618626+0000) 2024-02-19T20:22:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:02.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:02.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:02.701+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:02.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:02.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:03.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:02.937+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.938051+0000) 2024-02-19T20:22:03.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:02.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:03.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:02.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:03.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:02.809+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:03.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:02.845+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.845737+0000) 2024-02-19T20:22:03.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:03.208+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:03.208+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:03.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:03.208+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:03.806 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:03.594+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58721 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:03.807 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:03.588+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.589782+0000) 2024-02-19T20:22:03.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:03.588+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.589868+0000) 2024-02-19T20:22:03.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:03.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:03.807 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:03.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:03.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:03.592+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.593560+0000) 2024-02-19T20:22:03.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:03.617+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.618807+0000) 2024-02-19T20:22:03.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:03.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:03.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:03.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:03.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:03.707+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:03.808 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:03.707+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:04.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:03.937+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.938158+0000) 2024-02-19T20:22:04.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:03.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:04.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:03.844+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.845874+0000) 2024-02-19T20:22:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:04.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:04.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:04.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:04.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:04.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:04.620+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.621166+0000) 2024-02-19T20:22:04.764 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:04.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:04.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:04.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:04.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:04.738+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:04.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:04.738+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:05.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:04.937+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.938338+0000) 2024-02-19T20:22:05.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:04.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:05.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:04.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:05.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:04.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:05.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:04.845+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.846022+0000) 2024-02-19T20:22:05.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:05.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:05.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:05.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:05.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:05.245+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:05.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:05.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:05.846 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:05.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:05.847 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:05.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:05.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:05.623+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.624050+0000) 2024-02-19T20:22:05.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:05.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:05.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:05.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:05.847 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:05.740+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:05.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:05.740+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:06.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:05.845+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.846228+0000) 2024-02-19T20:22:06.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:05.937+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.938550+0000) 2024-02-19T20:22:06.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:06.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:06.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:06.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:06.292+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:06.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:06.626+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.626928+0000) 2024-02-19T20:22:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:06.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:06.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:06.743+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:06.814 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:06.743+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:07.167 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:06.845+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.846430+0000) 2024-02-19T20:22:07.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:06.938+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.938735+0000) 2024-02-19T20:22:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:07.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:07.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:07.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:07.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:07.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:07.625+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.627256+0000) 2024-02-19T20:22:07.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:07.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:07.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:07.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:07.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:07.712+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:07.767 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:07.712+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:08.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:07.937+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.938875+0000) 2024-02-19T20:22:08.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:07.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:08.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:07.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:08.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:07.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:08.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:07.845+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.846599+0000) 2024-02-19T20:22:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:08.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:08.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:08.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:08.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:08.766 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:08.603+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58795 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:08.767 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:08.597+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.598481+0000) 2024-02-19T20:22:08.767 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:08.596+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.598351+0000) 2024-02-19T20:22:08.767 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:08.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:08.767 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:08.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:08.767 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:08.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:08.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:08.601+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.603147+0000) 2024-02-19T20:22:08.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:08.626+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.627471+0000) 2024-02-19T20:22:08.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:08.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:08.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:08.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:08.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:08.734+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:08.768 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:08.734+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:09.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:08.938+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.939043+0000) 2024-02-19T20:22:09.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:08.846+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.846707+0000) 2024-02-19T20:22:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:09.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:09.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:09.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:09.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:09.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:09.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:09.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:09.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:09.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:09.742+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:09.786 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:09.629+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.629850+0000) 2024-02-19T20:22:09.786 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:09.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:09.787 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:09.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:10.160 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:09.846+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.846849+0000) 2024-02-19T20:22:10.160 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:09.938+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.939201+0000) 2024-02-19T20:22:10.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:09.785+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:10.161 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:09.785+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:10.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:10.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:10.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:10.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:10.275+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:10.757 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:10.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:10.757 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:10.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:10.757 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:10.751+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:10.758 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:10.632+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.632781+0000) 2024-02-19T20:22:10.758 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:10.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:10.758 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:10.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:11.009 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:10.846+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.847055+0000) 2024-02-19T20:22:11.009 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:10.938+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.939410+0000) 2024-02-19T20:22:11.010 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:10.756+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:11.010 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:10.757+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:11.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:11.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:11.635 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:11.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:11.636 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:11.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:11.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:11.634+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.635412+0000) 2024-02-19T20:22:11.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:11.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:11.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:11.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:11.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:11.797+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:11.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:11.797+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:11.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:11.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:11.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:11.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:11.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:11.716+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:11.941 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:11.846+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.847289+0000) 2024-02-19T20:22:12.212 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:11.939+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.939615+0000) 2024-02-19T20:22:12.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:12.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:12.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:12.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:12.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:12.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:12.635+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.635883+0000) 2024-02-19T20:22:12.827 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:12.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:12.827 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:12.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:12.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:12.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:12.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:12.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:13.107 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:12.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.847493+0000) 2024-02-19T20:22:13.108 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:12.939+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.939754+0000) 2024-02-19T20:22:13.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:12.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:13.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:12.826+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:13.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:12.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:13.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:13.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:13.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:13.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:13.257+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:13.823 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:13.613+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58867 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:13.824 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:13.607+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.607500+0000) 2024-02-19T20:22:13.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:13.606+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.607213+0000) 2024-02-19T20:22:13.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:13.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:13.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:13.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:13.825 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:13.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:13.825 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:13.610+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.611420+0000) 2024-02-19T20:22:13.825 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:13.635+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.636135+0000) 2024-02-19T20:22:13.825 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:13.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:13.825 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:13.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:13.826 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:13.822+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:13.826 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:13.823+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:14.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:13.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.847675+0000) 2024-02-19T20:22:14.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:13.939+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.939856+0000) 2024-02-19T20:22:14.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:14.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:14.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:14.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:14.261+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:14.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:14.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:14.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:14.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:14.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:14.651+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:14.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:14.638+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.638641+0000) 2024-02-19T20:22:14.843 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:14.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:15.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:14.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.847771+0000) 2024-02-19T20:22:15.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:14.939+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.940001+0000) 2024-02-19T20:22:15.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:14.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:15.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:14.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:15.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:14.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:15.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:15.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:15.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:15.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:15.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:15.837 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:15.640+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.641334+0000) 2024-02-19T20:22:15.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:15.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:15.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:15.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:15.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:15.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:15.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:15.620+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:16.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:15.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.847917+0000) 2024-02-19T20:22:16.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:15.939+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.940195+0000) 2024-02-19T20:22:16.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:15.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:16.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:15.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:16.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:15.837+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:16.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:16.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:16.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:16.330+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:16.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:16.643+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.644226+0000) 2024-02-19T20:22:16.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:16.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:16.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:16.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:16.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:16.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:17.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:16.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.848086+0000) 2024-02-19T20:22:17.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:16.940+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.940372+0000) 2024-02-19T20:22:17.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:16.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:17.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:16.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:17.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:16.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:17.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:16.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:17.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:17.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:17.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:17.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:17.350+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:17.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:17.644+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.644716+0000) 2024-02-19T20:22:17.824 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:17.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:17.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:17.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:17.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:17.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:17.824 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:17.628+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:18.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:17.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.848259+0000) 2024-02-19T20:22:18.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:17.940+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.940617+0000) 2024-02-19T20:22:18.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:17.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:18.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:17.823+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:18.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:17.823+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:18.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:18.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:18.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:18.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:18.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:18.829 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:18.621+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 58940 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:18.829 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:18.615+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.616353+0000) 2024-02-19T20:22:18.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:18.618+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.618958+0000) 2024-02-19T20:22:18.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:18.644+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.645009+0000) 2024-02-19T20:22:18.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:18.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:18.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:18.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:18.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:18.614+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.615183+0000) 2024-02-19T20:22:18.830 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:18.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:18.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:18.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:18.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:18.673+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:19.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:18.940+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.940814+0000) 2024-02-19T20:22:19.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:18.847+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.848371+0000) 2024-02-19T20:22:19.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:18.828+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:19.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:18.828+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:19.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:19.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:19.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:19.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:19.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:19.365+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:19.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:19.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:19.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:19.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:19.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:19.650+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:19.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:19.648+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.648685+0000) 2024-02-19T20:22:19.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:19.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:19.836 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:19.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:20.161 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:19.848+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.848523+0000) 2024-02-19T20:22:20.162 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:19.940+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.940962+0000) 2024-02-19T20:22:20.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:19.834+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:20.162 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:19.835+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:20.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:20.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:20.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:20.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:20.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:20.848 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:20.652+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.652551+0000) 2024-02-19T20:22:20.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:20.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:20.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:20.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:20.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:20.683+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:21.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:20.848+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.848730+0000) 2024-02-19T20:22:21.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:20.940+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.941137+0000) 2024-02-19T20:22:21.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:20.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:21.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:20.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:21.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:20.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:21.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:20.869+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:21.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:21.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:21.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:21.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:21.361+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:21.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:21.655+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.655312+0000) 2024-02-19T20:22:21.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:21.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:21.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:21.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:21.849 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:21.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:22.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:21.848+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.848939+0000) 2024-02-19T20:22:22.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:21.940+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.941321+0000) 2024-02-19T20:22:22.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:21.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:22.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:21.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:22.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:21.856+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:22.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:21.856+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:22.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:22.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:22.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:22.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:22.353+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:22.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:22.655+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.655731+0000) 2024-02-19T20:22:22.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:22.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:22.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:22.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:22.840 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:22.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:22.840 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:22.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:23.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:22.848+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.849115+0000) 2024-02-19T20:22:23.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:22.941+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.941491+0000) 2024-02-19T20:22:23.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:22.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:23.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:22.838+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:23.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:22.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:23.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:23.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:23.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:23.390+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:23.830 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:23.624+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.625074+0000) 2024-02-19T20:22:23.831 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:23.630+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59014 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:23.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:23.626+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.627413+0000) 2024-02-19T20:22:23.831 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:23.655+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.656033+0000) 2024-02-19T20:22:23.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:23.623+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.623911+0000) 2024-02-19T20:22:23.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:23.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:23.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:23.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:23.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:23.726+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:24.171 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:23.848+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.849254+0000) 2024-02-19T20:22:24.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:23.941+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.941661+0000) 2024-02-19T20:22:24.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:23.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:24.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:23.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:24.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:23.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:24.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:23.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:24.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:24.658+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.658490+0000) 2024-02-19T20:22:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:24.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:24.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:24.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:24.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:24.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:24.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:24.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:24.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:24.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:24.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:25.171 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:24.941+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.941786+0000) 2024-02-19T20:22:25.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:24.849+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.849407+0000) 2024-02-19T20:22:25.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:24.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:25.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:24.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:25.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:24.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:25.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:24.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:25.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:25.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:25.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:25.402+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:25.849 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:25.660+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.661184+0000) 2024-02-19T20:22:25.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:25.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:25.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:25.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:25.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:25.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:26.171 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:25.849+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.849534+0000) 2024-02-19T20:22:26.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:25.941+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.941959+0000) 2024-02-19T20:22:26.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:25.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:26.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:25.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:26.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:25.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:26.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:25.949+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:26.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:26.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:26.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:26.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:26.355+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:26.813 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:26.664+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.664195+0000) 2024-02-19T20:22:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:26.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:26.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:26.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:26.617+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:27.171 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:26.849+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.849712+0000) 2024-02-19T20:22:27.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:26.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.942189+0000) 2024-02-19T20:22:27.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:26.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:27.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:26.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:27.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:26.947+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:27.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:26.947+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:27.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:27.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:27.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:27.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:27.312+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:27.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:27.664+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.664695+0000) 2024-02-19T20:22:27.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:27.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:27.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:27.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:27.850 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:27.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:28.171 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:27.849+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.849892+0000) 2024-02-19T20:22:28.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:27.941+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.942353+0000) 2024-02-19T20:22:28.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:27.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:28.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:27.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:28.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:27.936+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:28.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:27.937+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:28.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:28.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:28.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:28.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:28.850 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:28.639+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59087 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:28.850 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:28.632+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.633285+0000) 2024-02-19T20:22:28.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:28.635+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.636219+0000) 2024-02-19T20:22:28.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:28.664+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.664901+0000) 2024-02-19T20:22:28.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:28.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:28.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:28.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:28.852 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:28.601+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:28.852 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:28.632+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.633024+0000) 2024-02-19T20:22:29.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:28.849+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.850095+0000) 2024-02-19T20:22:29.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:28.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.942533+0000) 2024-02-19T20:22:29.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:28.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:29.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:28.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:29.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:28.981+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:29.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:28.982+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:29.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:29.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:29.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:29.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:29.425 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:29.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:29.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:29.666+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.667423+0000) 2024-02-19T20:22:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:29.623+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:29.623+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:29.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:29.623+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:30.162 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:29.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.850255+0000) 2024-02-19T20:22:30.163 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:29.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.942707+0000) 2024-02-19T20:22:30.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:29.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:30.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:29.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:30.163 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:29.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:30.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:29.971+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:30.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:30.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:30.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:30.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:30.850 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:30.669+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.670262+0000) 2024-02-19T20:22:30.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:30.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:30.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:30.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:30.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:30.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:31.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:30.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.850439+0000) 2024-02-19T20:22:31.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:30.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.942880+0000) 2024-02-19T20:22:31.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:31.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:31.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:31.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:31.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:31.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:31.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:31.017+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:31.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:31.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:31.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:31.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:31.315+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:31.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:31.673+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.673198+0000) 2024-02-19T20:22:31.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:31.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:31.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:31.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:31.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:31.635+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:32.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:31.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.850673+0000) 2024-02-19T20:22:32.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:31.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.943007+0000) 2024-02-19T20:22:32.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:32.059+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:32.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:32.059+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:32.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:32.059+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:32.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:32.059+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:32.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:32.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:32.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:32.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:32.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:32.673+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.673690+0000) 2024-02-19T20:22:32.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:32.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:32.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:32.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:32.851 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:32.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:33.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:32.943+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.943184+0000) 2024-02-19T20:22:33.173 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:32.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.850875+0000) 2024-02-19T20:22:33.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:33.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:33.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:33.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.011+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:33.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:33.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:33.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:33.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:33.360+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:33.831 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:33.646+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59160 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:33.831 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:33.640+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.641812+0000) 2024-02-19T20:22:33.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.643+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.645052+0000) 2024-02-19T20:22:33.832 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.672+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.673884+0000) 2024-02-19T20:22:33.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:33.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:33.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:33.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:33.833 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:33.572+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:33.833 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:33.640+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.642276+0000) 2024-02-19T20:22:34.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:33.849+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.851036+0000) 2024-02-19T20:22:34.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:33.941+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.943358+0000) 2024-02-19T20:22:34.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:34.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:34.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.994+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:34.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:33.994+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:34.467 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:34.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:34.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:34.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:34.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:34.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:34.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:34.676+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.677484+0000) 2024-02-19T20:22:34.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:34.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:34.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:34.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:34.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:35.173 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:34.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.851193+0000) 2024-02-19T20:22:35.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:34.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.943528+0000) 2024-02-19T20:22:35.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:35.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:35.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:35.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:35.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:35.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:35.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:35.028+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:35.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:35.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:35.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:35.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:35.384+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:35.851 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:35.678+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.680182+0000) 2024-02-19T20:22:35.852 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:35.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:35.852 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:35.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:35.852 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:35.565+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:36.173 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:35.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.851431+0000) 2024-02-19T20:22:36.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:35.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.943712+0000) 2024-02-19T20:22:36.173 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:36.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:36.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:36.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:36.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:36.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:36.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:36.077+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:36.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:36.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:36.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:36.478 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:36.391+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:36.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:36.681+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.682817+0000) 2024-02-19T20:22:36.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:36.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:36.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:36.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:36.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:36.551+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:37.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:36.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.851629+0000) 2024-02-19T20:22:37.105 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:36.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.943886+0000) 2024-02-19T20:22:37.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:37.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:37.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:37.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:37.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:37.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:37.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:37.104+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:37.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:37.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:37.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:37.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:37.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:37.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:37.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:37.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:37.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:37.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:37.684 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:37.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:37.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:37.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.851820+0000) 2024-02-19T20:22:37.944 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:37.681+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.683296+0000) 2024-02-19T20:22:38.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:37.943+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.944092+0000) 2024-02-19T20:22:38.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:38.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:38.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:38.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:38.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:38.117+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:38.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:38.117+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:38.528 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:38.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:38.528 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:38.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:38.528 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:38.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:38.528 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:38.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:38.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:38.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:38.778 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:38.655+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59234 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:38.779 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:38.647+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.649317+0000) 2024-02-19T20:22:38.779 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:38.648+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.650241+0000) 2024-02-19T20:22:38.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:38.651+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.652833+0000) 2024-02-19T20:22:38.779 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:38.682+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.683574+0000) 2024-02-19T20:22:39.120 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:38.850+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.852011+0000) 2024-02-19T20:22:39.121 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:38.942+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.944280+0000) 2024-02-19T20:22:39.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:39.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:39.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:39.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:39.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:39.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:39.442 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:39.119+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:39.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:39.684+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.686019+0000) 2024-02-19T20:22:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:39.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:39.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:39.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:39.440+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:39.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:39.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:39.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:39.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:40.163 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:39.851+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.852184+0000) 2024-02-19T20:22:40.164 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:39.943+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.944400+0000) 2024-02-19T20:22:40.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:40.087+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:40.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:40.087+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:40.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:40.087+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:40.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:40.087+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:40.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:40.687+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.688788+0000) 2024-02-19T20:22:40.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:40.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:40.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:40.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:40.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:40.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:40.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:40.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:40.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:40.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:40.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:41.015 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:40.851+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.852338+0000) 2024-02-19T20:22:41.015 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:40.943+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.944641+0000) 2024-02-19T20:22:41.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:41.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:41.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:41.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:41.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:41.118+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:41.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:41.118+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:41.692 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:41.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:41.692 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:41.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:41.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:41.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:41.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:41.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:41.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:41.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:41.693 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:41.437+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:41.946 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:41.851+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.852520+0000) 2024-02-19T20:22:41.946 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:41.690+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.691465+0000) 2024-02-19T20:22:42.240 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:41.943+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.944846+0000) 2024-02-19T20:22:42.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:42.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:42.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:42.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:42.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:42.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:42.241 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:42.099+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:42.532 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:42.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:42.532 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:42.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:42.532 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:42.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:42.533 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:42.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:42.533 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:42.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:42.533 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:42.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:42.853 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:42.690+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.691897+0000) 2024-02-19T20:22:43.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:42.851+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.852697+0000) 2024-02-19T20:22:43.174 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:42.943+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.944989+0000) 2024-02-19T20:22:43.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:43.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:43.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:43.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:43.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:43.065+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:43.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:43.065+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:43.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:43.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:43.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:43.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:43.451 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:43.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:43.727 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:43.663+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59307 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:43.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:43.660+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.661730+0000) 2024-02-19T20:22:43.728 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:43.690+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.692168+0000) 2024-02-19T20:22:43.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:43.657+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.659090+0000) 2024-02-19T20:22:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:43.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:43.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:43.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:43.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:43.656+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.657833+0000) 2024-02-19T20:22:44.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:43.852+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.852856+0000) 2024-02-19T20:22:44.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:43.944+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.945193+0000) 2024-02-19T20:22:44.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:44.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:44.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:44.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:44.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:44.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:44.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:44.061+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:44.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:44.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:44.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:44.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:44.695 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:44.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:44.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:44.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:44.695 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:44.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:44.696 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:44.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:44.945 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:44.851+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.852971+0000) 2024-02-19T20:22:44.945 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:44.693+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.694602+0000) 2024-02-19T20:22:45.227 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:44.944+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.945317+0000) 2024-02-19T20:22:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:45.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:45.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:45.228 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:45.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:45.229 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:45.072+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:45.535 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:45.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:45.536 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:45.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:45.536 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:45.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:45.536 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:45.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:45.537 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:45.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:45.537 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:45.431+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:45.853 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:45.696+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.697484+0000) 2024-02-19T20:22:46.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:45.852+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.853167+0000) 2024-02-19T20:22:46.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:45.944+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.945479+0000) 2024-02-19T20:22:46.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:46.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:46.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:46.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:46.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:46.072+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:46.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:46.072+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:46.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:46.699+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.700702+0000) 2024-02-19T20:22:46.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:46.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:46.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:46.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:46.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:46.476+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:46.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:46.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:46.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:46.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:46.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:46.471+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:47.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:46.851+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.853329+0000) 2024-02-19T20:22:47.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:46.944+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.945641+0000) 2024-02-19T20:22:47.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:47.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:47.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:47.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:47.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:47.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:47.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:47.061+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:47.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:47.700+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.701390+0000) 2024-02-19T20:22:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:47.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:47.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:47.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:47.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:47.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:47.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:47.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:47.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:47.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:48.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:47.852+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.853508+0000) 2024-02-19T20:22:48.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:47.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.945847+0000) 2024-02-19T20:22:48.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:48.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:48.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:48.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:48.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:48.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:48.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:48.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:48.794 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:48.672+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59381 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:48.795 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:48.668+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.669660+0000) 2024-02-19T20:22:48.795 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:48.700+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.701675+0000) 2024-02-19T20:22:48.795 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:48.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:48.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:48.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:48.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:48.513+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:48.796 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:48.665+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.666796+0000) 2024-02-19T20:22:48.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:48.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:48.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:48.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:48.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:48.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:48.797 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:48.665+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.666960+0000) 2024-02-19T20:22:49.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:48.852+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.853706+0000) 2024-02-19T20:22:49.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:48.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.946024+0000) 2024-02-19T20:22:49.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:49.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:49.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:49.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:49.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:49.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:49.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:49.049+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:49.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:49.703+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.704011+0000) 2024-02-19T20:22:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:49.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:49.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:49.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:49.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:49.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:49.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:49.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:49.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:50.164 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:49.852+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.853835+0000) 2024-02-19T20:22:50.164 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:49.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.946189+0000) 2024-02-19T20:22:50.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:50.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:50.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:50.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:50.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:50.011+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:50.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:50.011+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:50.727 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:50.705+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.706604+0000) 2024-02-19T20:22:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:50.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:50.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:50.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:50.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:50.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:50.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:50.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:50.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:50.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:51.175 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:50.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.946395+0000) 2024-02-19T20:22:51.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:50.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.853992+0000) 2024-02-19T20:22:51.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:51.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:51.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:51.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:51.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:51.032+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:51.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:51.032+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:51.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:51.708+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.709210+0000) 2024-02-19T20:22:51.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:51.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:51.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:51.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:51.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:51.541+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:51.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:51.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:51.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:51.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:51.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:51.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:52.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:51.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.854158+0000) 2024-02-19T20:22:52.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:51.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.946532+0000) 2024-02-19T20:22:52.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:52.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:52.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:52.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:52.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.708+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.709626+0000) 2024-02-19T20:22:52.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:52.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:52.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:52.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:52.854 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:52.586+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:52.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:52.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:52.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:52.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:52.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:52.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:53.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:52.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.854281+0000) 2024-02-19T20:22:53.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:52.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.946632+0000) 2024-02-19T20:22:53.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:53.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:53.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:53.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:52.989+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:53.802 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:53.680+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59454 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:53.802 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:53.676+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.678284+0000) 2024-02-19T20:22:53.803 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:53.708+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.709823+0000) 2024-02-19T20:22:53.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:53.563+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:53.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:53.563+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:53.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:53.563+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:53.803 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:53.673+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.675347+0000) 2024-02-19T20:22:53.804 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:53.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:53.804 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:53.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:53.804 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:53.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:53.804 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:53.673+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.675099+0000) 2024-02-19T20:22:54.176 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:53.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.854470+0000) 2024-02-19T20:22:54.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:53.946+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.946802+0000) 2024-02-19T20:22:54.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:53.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:54.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:53.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:54.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:53.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:54.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:53.963+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:54.854 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:54.710+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.712006+0000) 2024-02-19T20:22:54.854 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:54.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:54.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:54.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:54.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:54.527+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:54.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:54.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:54.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:54.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:54.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:54.612+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:55.176 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:54.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.854616+0000) 2024-02-19T20:22:55.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:54.946+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.946977+0000) 2024-02-19T20:22:55.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:54.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:55.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:54.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:55.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:54.922+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:55.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:54.922+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:55.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:55.715+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.715642+0000) 2024-02-19T20:22:55.855 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:55.570+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:55.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:55.570+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:55.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:55.570+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:55.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:55.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:55.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:55.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:55.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:55.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:56.115 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:55.854+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.854795+0000) 2024-02-19T20:22:56.116 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:55.946+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.947174+0000) 2024-02-19T20:22:56.116 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:55.969+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:55.969+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:55.969+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:56.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:55.969+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:56.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:56.717+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.718350+0000) 2024-02-19T20:22:56.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:56.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:56.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:56.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:56.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:56.596+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:56.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:56.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:56.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:56.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:56.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:56.570+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:57.176 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:56.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.854938+0000) 2024-02-19T20:22:57.176 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:56.945+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.947381+0000) 2024-02-19T20:22:57.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:56.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:57.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:56.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:57.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:56.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:57.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:56.935+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:57.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:57.718+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.718785+0000) 2024-02-19T20:22:57.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:57.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:57.855 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:57.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:57.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:57.538+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:57.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:57.576+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:57.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:57.576+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:57.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:57.576+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:58.176 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:57.853+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.855158+0000) 2024-02-19T20:22:58.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:57.946+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.947537+0000) 2024-02-19T20:22:58.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:57.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:58.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:57.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:58.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:57.968+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:58.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:57.968+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:58.811 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:22:58.689+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59528 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:22:58.811 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:58.685+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.687321+0000) 2024-02-19T20:22:58.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:58.718+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.718975+0000) 2024-02-19T20:22:58.812 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:58.561+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:58.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:58.561+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:58.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:58.561+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:58.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:58.682+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.683899+0000) 2024-02-19T20:22:58.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:58.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:58.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:58.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:58.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:58.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:58.814 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:58.682+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.683722+0000) 2024-02-19T20:22:59.176 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:58.854+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.855352+0000) 2024-02-19T20:22:59.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:58.947+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.947735+0000) 2024-02-19T20:22:59.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:59.177 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:59.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:22:59.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:59.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.719+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.721270+0000) 2024-02-19T20:22:59.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:59.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:59.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:59.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:59.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:59.543+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:22:59.857 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:59.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:59.857 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:59.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:22:59.858 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:59.587+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:00.164 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:22:59.947+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.947847+0000) 2024-02-19T20:23:00.164 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:22:59.854+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.855514+0000) 2024-02-19T20:23:00.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:00.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:00.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:00.165 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:22:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:22:59.963+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:00.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:00.723+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.723782+0000) 2024-02-19T20:23:00.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:00.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:00.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:00.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:00.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:00.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:00.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:00.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:00.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:00.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:00.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:00.554+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:01.176 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:00.855+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.855699+0000) 2024-02-19T20:23:01.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:00.947+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.948038+0000) 2024-02-19T20:23:01.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:00.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:01.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:00.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:01.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:00.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:01.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:00.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:01.855 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:01.726+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.726813+0000) 2024-02-19T20:23:01.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:01.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:01.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:01.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:01.856 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:01.521+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:01.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:01.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:01.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:01.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:01.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:01.536+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:02.177 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:01.855+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.855830+0000) 2024-02-19T20:23:02.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:01.947+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.948261+0000) 2024-02-19T20:23:02.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:01.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:02.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:01.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:02.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:01.950+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:02.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:01.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:02.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:02.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:02.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:02.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:02.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:02.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:02.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:02.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:02.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:02.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:02.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:02.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:02.855+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.855999+0000) 2024-02-19T20:23:02.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:02.948+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.948460+0000) 2024-02-19T20:23:02.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:02.726+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.727328+0000) 2024-02-19T20:23:03.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:02.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:03.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:02.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:03.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:02.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:03.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:02.988+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:03.548 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:03.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:03.548 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:03.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:03.549 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:03.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:03.549 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:03.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:03.549 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:03.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:03.549 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:03.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:03.822 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:03.699+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59601 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:03.822 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:03.692+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.692793+0000) 2024-02-19T20:23:03.823 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:03.691+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.692411+0000) 2024-02-19T20:23:03.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:03.694+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.695297+0000) 2024-02-19T20:23:03.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:03.727+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.727537+0000) 2024-02-19T20:23:04.177 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:03.855+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.856150+0000) 2024-02-19T20:23:04.178 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:03.948+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.948607+0000) 2024-02-19T20:23:04.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:03.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:04.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:03.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:04.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:03.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:04.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:03.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:04.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:04.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:04.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:04.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:04.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:04.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:04.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:04.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:04.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:04.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:04.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:04.495+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:04.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:04.855+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.856301+0000) 2024-02-19T20:23:04.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:04.729+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.729978+0000) 2024-02-19T20:23:04.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:04.948+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.948741+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:04.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:04.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:04.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:04.985+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:05.543 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:05.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:05.544 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:05.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:05.544 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:05.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:05.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:05.732+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.732569+0000) 2024-02-19T20:23:05.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:05.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:05.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:05.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:05.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:05.542+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:06.177 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:05.856+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.856502+0000) 2024-02-19T20:23:06.178 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:05.948+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.948911+0000) 2024-02-19T20:23:06.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:06.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:06.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:06.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.029+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:06.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:06.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:06.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:06.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:06.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:06.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:06.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.734+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.735295+0000) 2024-02-19T20:23:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:06.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:06.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:06.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:06.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:07.178 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:06.856+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.856711+0000) 2024-02-19T20:23:07.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:06.948+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.949103+0000) 2024-02-19T20:23:07.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:07.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:07.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:07.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:06.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:07.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:07.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:07.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:07.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:07.856 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:07.735+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.735836+0000) 2024-02-19T20:23:07.856 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:07.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:07.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:07.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:07.857 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:07.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:08.178 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:07.856+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.856842+0000) 2024-02-19T20:23:08.178 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:07.948+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.949280+0000) 2024-02-19T20:23:08.178 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:08.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:08.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:08.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:08.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:08.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:08.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:08.008+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:08.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:08.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:08.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:08.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:08.829 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:08.708+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59675 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:08.830 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:08.701+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.702296+0000) 2024-02-19T20:23:08.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:08.705+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.706176+0000) 2024-02-19T20:23:08.830 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:08.735+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.736097+0000) 2024-02-19T20:23:08.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:08.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:08.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:08.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:08.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:08.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:08.831 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:08.701+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.701896+0000) 2024-02-19T20:23:09.178 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:08.856+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.856940+0000) 2024-02-19T20:23:09.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:08.949+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.949513+0000) 2024-02-19T20:23:09.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:09.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:09.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:09.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:09.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:09.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:09.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:09.028+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:09.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:09.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:09.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:09.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:09.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:09.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:09.762 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:09.738+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.738516+0000) 2024-02-19T20:23:09.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:09.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:09.762 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:09.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:09.763 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:09.491+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:10.020 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:09.856+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.857046+0000) 2024-02-19T20:23:10.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:09.949+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.949684+0000) 2024-02-19T20:23:10.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:10.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:10.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:10.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:10.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:10.074+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:10.414 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:10.074+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:10.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:10.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:10.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:10.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:10.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:10.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:10.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:10.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:10.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:10.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:10.446+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:11.020 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:10.741+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.741324+0000) 2024-02-19T20:23:11.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:10.949+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.949831+0000) 2024-02-19T20:23:11.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:10.856+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.857233+0000) 2024-02-19T20:23:11.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:11.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:11.410 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:11.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:11.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:11.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:11.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:11.098+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:11.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:11.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:11.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:11.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:11.410+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:11.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:11.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:11.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:11.479+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:12.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:11.949+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.949990+0000) 2024-02-19T20:23:12.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:11.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.857416+0000) 2024-02-19T20:23:12.021 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:11.743+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.744005+0000) 2024-02-19T20:23:12.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:12.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:12.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:12.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:12.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:12.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:12.440 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:12.055+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:12.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:12.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:12.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:12.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:12.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:12.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:12.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:12.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:12.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:12.439+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:13.020 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:12.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.857643+0000) 2024-02-19T20:23:13.021 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:12.949+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.950147+0000) 2024-02-19T20:23:13.021 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:12.744+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.744507+0000) 2024-02-19T20:23:13.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:13.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:13.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:13.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:13.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:13.075+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:13.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:13.076+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:13.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:13.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:13.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:13.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:13.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:13.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:13.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:13.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:13.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:13.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:13.711 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:13.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:13.977 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:13.717+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59748 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:13.713+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.714202+0000) 2024-02-19T20:23:13.977 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:13.744+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.744720+0000) 2024-02-19T20:23:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:13.710+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.710938+0000) 2024-02-19T20:23:13.978 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:13.949+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.950343+0000) 2024-02-19T20:23:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:13.710+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.711278+0000) 2024-02-19T20:23:13.978 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:13.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.857796+0000) 2024-02-19T20:23:14.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:14.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:14.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:14.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:14.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:14.036+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:14.407 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:14.036+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:14.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:14.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:14.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:14.450+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:14.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:14.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:14.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:14.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:15.014 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:14.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.857907+0000) 2024-02-19T20:23:15.014 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:14.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.950531+0000) 2024-02-19T20:23:15.015 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:14.746+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.747183+0000) 2024-02-19T20:23:15.015 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:15.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:15.015 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:15.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:15.266 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:15.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:15.267 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:15.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:15.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:15.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:15.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:15.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:15.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:15.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:15.561 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:15.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:15.561 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:15.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:15.561 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:15.372+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:15.857 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:15.749+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.749852+0000) 2024-02-19T20:23:16.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:15.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.950679+0000) 2024-02-19T20:23:16.179 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:15.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.858094+0000) 2024-02-19T20:23:16.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:16.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:16.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:16.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:16.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:16.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:16.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:16.037+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:16.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:16.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:16.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:16.477 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:16.396+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:16.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:16.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:16.477 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:16.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:16.812 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:16.752+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.752568+0000) 2024-02-19T20:23:17.179 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:16.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.858220+0000) 2024-02-19T20:23:17.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:16.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.950887+0000) 2024-02-19T20:23:17.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:17.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:17.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:17.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:17.180 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:17.044+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:17.181 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:17.044+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:17.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:17.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:17.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:17.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:17.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:17.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:17.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:17.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:17.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:17.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:17.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:17.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:18.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:17.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.858415+0000) 2024-02-19T20:23:18.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:17.752+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.753130+0000) 2024-02-19T20:23:18.023 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:17.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.951101+0000) 2024-02-19T20:23:18.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:18.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:18.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:18.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:18.443 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:18.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:18.444 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:18.062+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:18.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:18.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:18.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:18.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:18.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:18.449+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:18.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:18.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:18.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:18.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:18.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:18.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:18.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:18.718+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.719231+0000) 2024-02-19T20:23:18.976 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:18.725+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59822 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:18.977 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:18.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.951230+0000) 2024-02-19T20:23:18.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:18.718+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.719409+0000) 2024-02-19T20:23:18.977 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:18.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.858589+0000) 2024-02-19T20:23:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:18.722+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.722490+0000) 2024-02-19T20:23:18.978 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:18.752+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.753390+0000) 2024-02-19T20:23:19.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:19.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:19.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:19.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:19.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:19.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:19.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:19.105+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:19.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:19.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:19.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:19.486+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:19.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:19.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:19.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:19.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:19.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:19.408+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:20.021 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:19.755+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.755708+0000) 2024-02-19T20:23:20.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:19.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.951338+0000) 2024-02-19T20:23:20.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:19.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.858690+0000) 2024-02-19T20:23:20.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:20.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:20.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:20.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:20.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:20.095+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:20.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:20.095+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:20.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:20.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:20.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:20.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:20.691 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:20.394+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:20.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:20.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:20.691 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:20.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:20.692 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:20.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:20.950 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:20.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.858847+0000) 2024-02-19T20:23:20.951 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:20.758+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.758431+0000) 2024-02-19T20:23:21.226 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:20.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.951559+0000) 2024-02-19T20:23:21.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:21.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:21.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:21.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:21.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:21.111+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:21.227 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:21.111+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:21.565 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:21.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:21.565 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:21.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:21.565 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:21.515+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:21.566 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:21.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:21.566 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:21.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:21.566 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:21.387+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:21.858 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:21.761+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.761228+0000) 2024-02-19T20:23:22.129 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:21.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.859029+0000) 2024-02-19T20:23:22.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:21.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.951718+0000) 2024-02-19T20:23:22.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:22.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:22.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:22.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:22.394 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:22.130+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:22.395 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:22.130+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:22.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:22.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:22.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:22.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:22.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:22.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:22.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:22.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:22.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:22.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:22.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:22.529+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:23.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:22.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.951860+0000) 2024-02-19T20:23:23.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:22.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.859214+0000) 2024-02-19T20:23:23.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:22.761+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.761789+0000) 2024-02-19T20:23:23.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:23.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:23.382 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:23.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:23.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:23.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:23.383 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:23.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:23.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:23.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:23.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:23.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:23.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:23.501+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:23.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:23.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:23.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:23.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:23.383+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:23.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:23.726+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.727614+0000) 2024-02-19T20:23:24.022 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:23.733+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59895 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:24.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:23.730+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.732234+0000) 2024-02-19T20:23:24.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:23.760+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.762000+0000) 2024-02-19T20:23:24.023 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:23.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.952029+0000) 2024-02-19T20:23:24.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:23.726+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.727983+0000) 2024-02-19T20:23:24.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:23.857+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.859383+0000) 2024-02-19T20:23:24.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:24.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:24.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:24.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:24.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:24.115+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:24.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:24.115+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:24.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:24.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:24.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:24.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:24.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:24.344+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:24.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:24.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:24.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:24.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:24.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:24.514+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:25.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:24.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.952161+0000) 2024-02-19T20:23:25.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:24.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.859551+0000) 2024-02-19T20:23:25.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:24.763+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.764579+0000) 2024-02-19T20:23:25.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:25.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:25.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:25.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:25.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:25.099+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:25.368 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:25.099+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:25.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:25.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:25.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:25.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:25.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:25.368+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:25.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:25.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:25.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:25.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:25.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:25.534+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:26.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:25.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.859732+0000) 2024-02-19T20:23:26.023 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:25.950+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.952373+0000) 2024-02-19T20:23:26.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:25.765+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.767330+0000) 2024-02-19T20:23:26.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:26.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:26.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:26.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:26.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:26.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:26.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:26.148+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:26.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:26.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:26.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:26.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:26.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:26.525+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:26.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:26.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:26.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:26.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:26.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:26.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:27.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:26.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.859942+0000) 2024-02-19T20:23:27.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:26.769+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.770004+0000) 2024-02-19T20:23:27.023 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:26.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.952475+0000) 2024-02-19T20:23:27.427 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:27.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:27.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:27.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:27.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:27.194+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:27.428 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:27.194+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:27.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:27.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:27.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:27.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:27.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:27.478+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:27.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:27.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:27.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:27.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:27.728 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:27.427+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:28.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:27.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.860090+0000) 2024-02-19T20:23:28.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:27.769+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.770543+0000) 2024-02-19T20:23:28.023 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:27.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.952673+0000) 2024-02-19T20:23:28.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:28.225+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:28.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:28.225+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:28.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:28.225+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:28.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:28.225+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:28.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:28.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:28.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:28.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:28.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:28.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:28.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:28.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:28.429+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:28.742+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 59969 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:29.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:28.735+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.736985+0000) 2024-02-19T20:23:29.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:28.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.860260+0000) 2024-02-19T20:23:29.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:28.739+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.740740+0000) 2024-02-19T20:23:29.024 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:28.769+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.770809+0000) 2024-02-19T20:23:29.024 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:28.735+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.736992+0000) 2024-02-19T20:23:29.024 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:28.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.952863+0000) 2024-02-19T20:23:29.433 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:29.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:29.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:29.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:29.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:29.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:29.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:29.217+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:29.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:29.517+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:29.517+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:29.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:29.517+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:29.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:29.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:29.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:29.433+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:30.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:29.858+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.860397+0000) 2024-02-19T20:23:30.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:29.771+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.773127+0000) 2024-02-19T20:23:30.023 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:29.951+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.952981+0000) 2024-02-19T20:23:30.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:30.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:30.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:30.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:30.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:30.168+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:30.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:30.168+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:30.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:30.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:30.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:30.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:30.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:30.487+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:30.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:30.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:30.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:30.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:30.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:30.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:31.181 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:30.859+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.860612+0000) 2024-02-19T20:23:31.181 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:30.774+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.775724+0000) 2024-02-19T20:23:31.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:30.952+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.953168+0000) 2024-02-19T20:23:31.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:31.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:31.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:31.209+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:31.446 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:31.209+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:31.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:31.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:31.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:31.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:31.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:31.446+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:31.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:31.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:31.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:31.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:31.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:31.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:32.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:31.859+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.860788+0000) 2024-02-19T20:23:32.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:31.777+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.778510+0000) 2024-02-19T20:23:32.183 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:31.952+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.953326+0000) 2024-02-19T20:23:32.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:32.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:32.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:32.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:32.459 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:32.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:32.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:32.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:32.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:32.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:32.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:32.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:32.460 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:32.219+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:32.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:32.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:32.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:32.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:32.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:32.459+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:33.181 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:32.860+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.860967+0000) 2024-02-19T20:23:33.182 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:32.778+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.779101+0000) 2024-02-19T20:23:33.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:32.952+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.953541+0000) 2024-02-19T20:23:33.456 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:33.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:33.457 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:33.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:33.457 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:33.357+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:33.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:33.249+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:33.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:33.249+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:33.458 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:33.249+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:33.459 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:33.249+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:33.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:33.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:33.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:33.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:33.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:33.456+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:34.023 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:33.751+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60042 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:34.024 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:33.744+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.745889+0000) 2024-02-19T20:23:34.024 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:33.859+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.861124+0000) 2024-02-19T20:23:34.024 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:33.748+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.749873+0000) 2024-02-19T20:23:34.024 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:33.777+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.779361+0000) 2024-02-19T20:23:34.025 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:33.744+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.746149+0000) 2024-02-19T20:23:34.025 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:33.952+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.953672+0000) 2024-02-19T20:23:34.389 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:34.238+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:34.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:34.238+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:34.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:34.238+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:34.390 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:34.238+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:34.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:34.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:34.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:34.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:34.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:34.483+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:34.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:34.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:34.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:34.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:35.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:34.859+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.861280+0000) 2024-02-19T20:23:35.182 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:34.780+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.781717+0000) 2024-02-19T20:23:35.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:34.953+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.953835+0000) 2024-02-19T20:23:35.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:35.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:35.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:35.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:35.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:35.221+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:35.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:35.221+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:35.728 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:35.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:35.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:35.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:35.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:35.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:35.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:35.729 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:35.434+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:36.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:35.860+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.861466+0000) 2024-02-19T20:23:36.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:35.784+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.785516+0000) 2024-02-19T20:23:36.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:35.953+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.954035+0000) 2024-02-19T20:23:36.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:36.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:36.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:36.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:36.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:36.253+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:36.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:36.253+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:36.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:36.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:36.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:36.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:36.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:36.416+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:36.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:36.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:36.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:36.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:36.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:36.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:37.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:36.860+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.861643+0000) 2024-02-19T20:23:37.182 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:36.952+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.954231+0000) 2024-02-19T20:23:37.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:36.787+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.788267+0000) 2024-02-19T20:23:37.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:37.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:37.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:37.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:37.465 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:37.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:37.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:37.231+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:37.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:37.231+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:37.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:37.231+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:37.466 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:37.231+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:37.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:37.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:37.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:37.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:37.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:37.465+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:38.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:37.861+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.861837+0000) 2024-02-19T20:23:38.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:37.788+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.789030+0000) 2024-02-19T20:23:38.183 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:37.953+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.954387+0000) 2024-02-19T20:23:38.435 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:38.278+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:38.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:38.278+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:38.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:38.278+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:38.436 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:38.278+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:38.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:38.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:38.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:38.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:38.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:38.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:38.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:38.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:38.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:38.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:38.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:38.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:39.024 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:38.760+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60115 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:39.025 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:38.757+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.758600+0000) 2024-02-19T20:23:39.025 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:38.787+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.789378+0000) 2024-02-19T20:23:39.025 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:38.752+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.754373+0000) 2024-02-19T20:23:39.025 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:38.953+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.954545+0000) 2024-02-19T20:23:39.026 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:38.752+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.754085+0000) 2024-02-19T20:23:39.026 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:38.861+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.862014+0000) 2024-02-19T20:23:39.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:39.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:39.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:39.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:39.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:39.250+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:39.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:39.250+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:39.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:39.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:39.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:39.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:39.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:39.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:39.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:39.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:39.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:39.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:39.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:39.467+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:40.166 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:39.861+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.862167+0000) 2024-02-19T20:23:40.167 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:39.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.954711+0000) 2024-02-19T20:23:40.167 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:39.790+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.791900+0000) 2024-02-19T20:23:40.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:40.215+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:40.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:40.215+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:40.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:40.215+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:40.454 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:40.215+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:40.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:40.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:40.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:40.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:40.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:40.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:40.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:40.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:40.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:40.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:40.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:40.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:41.183 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:40.861+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.862359+0000) 2024-02-19T20:23:41.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:40.794+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.795604+0000) 2024-02-19T20:23:41.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:40.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.954883+0000) 2024-02-19T20:23:41.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:41.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:41.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:41.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:41.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:41.188+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:41.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:41.188+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:41.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:41.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:41.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:41.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:41.798 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:41.489+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:41.798 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:41.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:41.799 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:41.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:41.799 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:41.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:42.143 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:41.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.955044+0000) 2024-02-19T20:23:42.144 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:41.861+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.862555+0000) 2024-02-19T20:23:42.144 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:41.798+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.799415+0000) 2024-02-19T20:23:42.144 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:42.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:42.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:42.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:42.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:42.143+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:42.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:42.143+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:42.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:42.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:42.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:42.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:42.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:42.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:42.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:42.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:42.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:42.473+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:43.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:42.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.955216+0000) 2024-02-19T20:23:43.123 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:42.862+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.862733+0000) 2024-02-19T20:23:43.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:42.799+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.800171+0000) 2024-02-19T20:23:43.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:43.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:43.124 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:43.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:43.434 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:43.123+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:43.530 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:43.123+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:43.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:43.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:43.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:43.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:43.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:43.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:43.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:43.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:43.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:43.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:43.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:43.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:44.025 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:43.769+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60188 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:44.025 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:43.766+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.767783+0000) 2024-02-19T20:23:44.025 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:43.798+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.800387+0000) 2024-02-19T20:23:44.026 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:43.762+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.763592+0000) 2024-02-19T20:23:44.026 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:43.862+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.862955+0000) 2024-02-19T20:23:44.027 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:43.761+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.763329+0000) 2024-02-19T20:23:44.027 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:43.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.955424+0000) 2024-02-19T20:23:44.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:44.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:44.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:44.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:44.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:44.170+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:44.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:44.170+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:44.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:44.729 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:44.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:44.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:44.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:44.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:44.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:44.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:44.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:44.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:44.438+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:45.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:44.861+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.863126+0000) 2024-02-19T20:23:45.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:44.801+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.802730+0000) 2024-02-19T20:23:45.056 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:44.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.955532+0000) 2024-02-19T20:23:45.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:45.132+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:45.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:45.132+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:45.411 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:45.132+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:45.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:45.132+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:45.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:45.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:45.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:45.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:45.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:45.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:45.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:45.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:45.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:45.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:45.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:45.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:46.084 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:45.862+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.863312+0000) 2024-02-19T20:23:46.085 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:45.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.955657+0000) 2024-02-19T20:23:46.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:45.804+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.805364+0000) 2024-02-19T20:23:46.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:46.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:46.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:46.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:46.085 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:46.085+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:46.086 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:46.085+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:46.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:46.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:46.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:46.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:46.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:46.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:46.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:46.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:46.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:46.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:47.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:46.862+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.863505+0000) 2024-02-19T20:23:47.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:46.955+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.955855+0000) 2024-02-19T20:23:47.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:46.807+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.808225+0000) 2024-02-19T20:23:47.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:47.053+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:47.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:47.053+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:47.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:47.053+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:47.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:47.053+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:47.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:47.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:47.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:47.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:47.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:47.464+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:47.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:47.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:47.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:47.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:47.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:47.440+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:48.184 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:47.954+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.956047+0000) 2024-02-19T20:23:48.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:47.863+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.863685+0000) 2024-02-19T20:23:48.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:47.808+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.809091+0000) 2024-02-19T20:23:48.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:48.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:48.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:48.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:48.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:48.035+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:48.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:48.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:48.771 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:48.480+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:48.771 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:48.480+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:48.771 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:48.480+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:48.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:48.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:48.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:48.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:48.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:48.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:48.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:48.771+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.772895+0000) 2024-02-19T20:23:49.026 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:48.778+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60263 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:49.026 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:48.862+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.863827+0000) 2024-02-19T20:23:49.026 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:48.775+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.777098+0000) 2024-02-19T20:23:49.027 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:48.808+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.809431+0000) 2024-02-19T20:23:49.027 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:48.771+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.773112+0000) 2024-02-19T20:23:49.027 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:48.955+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.956210+0000) 2024-02-19T20:23:49.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:49.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:49.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:49.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:49.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:49.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:49.477 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:49.035+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:49.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:49.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:49.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:49.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:49.810 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:49.482+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:49.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:49.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:49.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:49.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:49.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:49.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:50.097 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:49.863+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.863983+0000) 2024-02-19T20:23:50.097 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:49.955+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.956356+0000) 2024-02-19T20:23:50.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:49.810+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.811832+0000) 2024-02-19T20:23:50.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:50.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:50.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:50.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:50.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:50.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:50.098 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:50.025+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:50.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:50.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:50.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:50.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:50.813 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:50.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:50.813 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:50.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:50.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:50.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:50.814 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:50.532+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:51.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:50.863+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.864129+0000) 2024-02-19T20:23:51.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:50.956+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.956509+0000) 2024-02-19T20:23:51.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:50.814+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.814671+0000) 2024-02-19T20:23:51.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:51.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:51.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:51.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:51.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:51.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:51.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:51.003+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:51.815 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:51.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:51.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:51.492+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:51.816 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:51.493+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:51.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:51.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:51.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:51.542+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:51.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:51.543+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:52.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:51.863+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.864312+0000) 2024-02-19T20:23:52.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:51.956+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.956673+0000) 2024-02-19T20:23:52.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:51.817+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.817435+0000) 2024-02-19T20:23:52.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:52.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:52.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:52.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:52.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:52.034+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:52.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:52.034+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:52.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:52.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:52.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:52.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:52.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:52.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:52.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:52.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:52.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:52.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:52.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:52.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:53.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:52.864+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.864478+0000) 2024-02-19T20:23:53.185 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:52.956+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.956869+0000) 2024-02-19T20:23:53.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:52.817+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.818141+0000) 2024-02-19T20:23:53.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:53.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:53.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:53.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:53.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:53.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:53.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:53.018+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:53.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:53.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:53.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:53.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:53.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:53.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:53.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:53.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:53.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:53.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:54.185 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:53.788+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60336 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:54.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:53.781+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.782126+0000) 2024-02-19T20:23:54.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:53.864+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.864625+0000) 2024-02-19T20:23:54.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:53.781+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.782332+0000) 2024-02-19T20:23:54.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:53.956+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.957044+0000) 2024-02-19T20:23:54.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:53.785+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.786289+0000) 2024-02-19T20:23:54.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:53.818+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.818465+0000) 2024-02-19T20:23:54.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:54.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:54.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:54.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:54.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:54.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:54.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:54.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:54.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:54.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:54.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:54.523+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:54.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:54.523+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:54.819 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:54.523+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:55.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:54.956+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.957204+0000) 2024-02-19T20:23:55.139 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:54.864+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.864748+0000) 2024-02-19T20:23:55.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.820+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.820730+0000) 2024-02-19T20:23:55.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.966+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:55.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.966+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:55.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.966+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:55.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:54.966+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:55.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:55.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:55.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:55.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:55.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:55.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:55.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:55.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:55.727 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:55.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:56.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:55.864+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.864950+0000) 2024-02-19T20:23:56.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:55.957+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.957412+0000) 2024-02-19T20:23:56.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:55.823+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.823590+0000) 2024-02-19T20:23:56.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:55.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:56.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:55.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:56.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:55.926+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:56.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:55.926+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:56.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:56.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:56.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:56.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:56.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:56.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:56.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:56.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:56.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:56.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:56.811 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:56.507+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:57.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:56.864+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.865112+0000) 2024-02-19T20:23:57.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:56.957+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.957621+0000) 2024-02-19T20:23:57.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:56.825+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.826318+0000) 2024-02-19T20:23:57.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:56.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:57.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:56.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:57.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:56.878+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:57.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:56.878+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:57.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:57.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:57.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:57.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:57.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:57.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:57.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:57.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:57.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:57.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:57.825 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:57.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:58.185 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:57.864+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.865289+0000) 2024-02-19T20:23:58.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:57.957+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.957740+0000) 2024-02-19T20:23:58.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:57.826+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.826970+0000) 2024-02-19T20:23:58.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:57.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:58.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:57.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:58.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:57.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:58.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:57.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:58.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:58.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:58.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:58.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:58.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:58.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:58.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:58.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:58.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:58.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:58.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:58.462+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:59.186 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:23:58.797+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60410 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:23:59.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:58.790+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.791415+0000) 2024-02-19T20:23:59.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:58.957+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.957901+0000) 2024-02-19T20:23:59.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:58.790+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.790768+0000) 2024-02-19T20:23:59.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:58.865+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.865481+0000) 2024-02-19T20:23:59.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:58.793+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.794325+0000) 2024-02-19T20:23:59.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:58.826+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.827243+0000) 2024-02-19T20:23:59.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:58.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:59.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:58.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:59.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:58.844+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:23:59.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:58.844+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:59.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:59.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:59.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:59.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:59.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:59.553+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:23:59.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:59.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:59.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:59.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:23:59.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:59.471+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:00.168 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:23:59.865+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.865622+0000) 2024-02-19T20:24:00.168 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:23:59.957+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.958030+0000) 2024-02-19T20:24:00.169 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:59.829+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.829980+0000) 2024-02-19T20:24:00.169 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:59.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:00.169 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:59.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:00.169 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:59.894+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:00.169 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:23:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:23:59.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:00.831 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:00.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:00.831 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:00.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:00.831 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:00.509+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:00.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:00.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:00.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:00.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:00.832 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:00.512+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:01.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:00.865+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.865806+0000) 2024-02-19T20:24:01.186 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:00.957+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.958217+0000) 2024-02-19T20:24:01.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:00.832+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.832901+0000) 2024-02-19T20:24:01.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:00.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:01.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:00.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:01.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:00.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:01.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:00.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:01.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:01.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:01.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:01.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:01.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:01.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:01.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:01.470+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:01.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:01.470+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:01.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:01.470+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:02.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:01.865+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.866016+0000) 2024-02-19T20:24:02.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:01.958+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.958354+0000) 2024-02-19T20:24:02.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:01.835+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.835919+0000) 2024-02-19T20:24:02.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:01.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:02.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:01.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:02.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:01.927+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:02.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:01.927+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:02.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:02.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:02.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:02.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:02.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:02.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:02.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:02.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:02.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:02.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:02.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:02.467+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:03.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:02.865+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.866166+0000) 2024-02-19T20:24:03.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:02.958+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.958497+0000) 2024-02-19T20:24:03.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:02.836+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.836682+0000) 2024-02-19T20:24:03.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:02.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:03.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:02.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:03.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:02.888+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:03.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:02.888+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:03.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:03.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:03.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:03.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:03.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:03.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:03.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:03.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:03.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:03.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:03.727 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:03.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:04.186 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:03.806+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60483 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:04.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:03.798+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.799422+0000) 2024-02-19T20:24:04.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:03.958+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.958680+0000) 2024-02-19T20:24:04.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:03.799+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.799525+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:03.865+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.866284+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:03.802+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.803370+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:03.836+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.836930+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:03.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:03.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:03.935+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:04.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:03.935+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:04.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:04.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:04.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:04.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:04.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:04.427+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:04.837 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:04.496+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:04.837 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:04.496+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:04.837 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:04.496+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:05.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:04.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.866439+0000) 2024-02-19T20:24:05.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:04.958+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.958885+0000) 2024-02-19T20:24:05.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:04.839+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.839453+0000) 2024-02-19T20:24:05.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:04.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:05.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:04.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:05.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:04.900+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:05.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:04.900+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:05.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:05.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:05.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:05.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:05.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:05.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:05.840 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:05.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:05.840 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:05.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:05.840 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:05.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:06.186 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:05.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.866608+0000) 2024-02-19T20:24:06.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:05.958+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.959079+0000) 2024-02-19T20:24:06.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:05.841+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.842185+0000) 2024-02-19T20:24:06.187 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:05.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:06.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:05.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:06.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:05.929+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:06.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:05.929+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:06.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:06.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:06.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:06.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:06.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:06.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:06.769 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:06.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:06.770 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:06.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:06.770 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:06.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:07.028 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:06.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.866792+0000) 2024-02-19T20:24:07.029 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:06.958+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.959236+0000) 2024-02-19T20:24:07.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:06.844+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.844966+0000) 2024-02-19T20:24:07.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:06.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:07.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:06.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:07.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:06.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:07.030 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:06.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:07.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:07.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:07.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:07.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:07.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:07.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:07.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:07.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:07.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:07.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:07.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:07.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:08.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:07.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.866955+0000) 2024-02-19T20:24:08.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:07.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.959480+0000) 2024-02-19T20:24:08.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:07.845+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.845708+0000) 2024-02-19T20:24:08.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:07.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:08.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:07.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:08.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:07.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:08.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:07.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:08.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:08.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:08.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:08.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:08.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:08.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:08.806 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:08.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:08.806 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:08.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:08.806 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:08.484+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:09.187 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:08.814+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60557 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:09.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:08.807+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.808145+0000) 2024-02-19T20:24:09.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:08.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.867144+0000) 2024-02-19T20:24:09.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:08.808+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.808485+0000) 2024-02-19T20:24:09.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:08.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.959666+0000) 2024-02-19T20:24:09.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:08.810+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.811112+0000) 2024-02-19T20:24:09.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:08.845+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.846038+0000) 2024-02-19T20:24:09.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:08.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:09.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:08.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:09.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:08.901+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:09.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:08.901+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:09.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:09.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:09.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:09.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:09.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:09.444+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:09.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:09.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:09.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:09.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:09.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:09.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:10.169 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:09.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.867296+0000) 2024-02-19T20:24:10.169 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:09.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.959778+0000) 2024-02-19T20:24:10.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:09.847+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.848341+0000) 2024-02-19T20:24:10.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:09.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:10.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:09.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:10.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:09.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:10.170 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:09.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:10.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:10.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:10.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:10.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:10.726 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:10.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:10.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:10.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:10.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:10.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:10.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:10.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:11.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:10.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.867526+0000) 2024-02-19T20:24:11.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:10.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.959928+0000) 2024-02-19T20:24:11.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:10.850+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.851028+0000) 2024-02-19T20:24:11.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:10.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:11.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:10.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:11.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:10.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:11.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:10.942+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:11.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:11.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:11.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:11.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:11.453 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:11.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:11.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:11.455+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:11.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:11.455+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:11.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:11.455+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:12.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:11.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.867684+0000) 2024-02-19T20:24:12.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:11.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.960133+0000) 2024-02-19T20:24:12.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:11.854+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.854887+0000) 2024-02-19T20:24:12.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:11.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:12.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:11.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:12.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:11.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:12.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:11.953+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:12.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:12.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:12.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:12.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:12.450 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:12.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:12.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:12.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:12.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:12.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:12.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:12.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:13.187 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:12.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.867875+0000) 2024-02-19T20:24:13.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:12.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.960296+0000) 2024-02-19T20:24:13.188 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:12.855+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.855666+0000) 2024-02-19T20:24:13.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:12.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:13.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:12.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:13.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:12.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:13.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:12.997+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:13.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:13.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:13.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:13.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:13.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:13.423+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:13.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:13.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:13.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:13.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:13.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:13.432+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:13.865 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:13.815+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.816498+0000) 2024-02-19T20:24:13.866 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:13.822+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60630 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:13.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:13.814+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.816334+0000) 2024-02-19T20:24:13.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:13.818+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.819932+0000) 2024-02-19T20:24:13.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:13.854+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.855866+0000) 2024-02-19T20:24:14.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:13.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.868056+0000) 2024-02-19T20:24:14.243 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:13.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.960434+0000) 2024-02-19T20:24:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:13.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:13.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:13.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:14.244 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:13.988+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:14.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:14.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:14.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:14.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:14.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:14.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:14.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:14.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:14.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:14.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:14.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:14.425+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:14.866 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:14.856+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.858251+0000) 2024-02-19T20:24:15.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:14.866+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.868223+0000) 2024-02-19T20:24:15.188 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:14.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.960584+0000) 2024-02-19T20:24:15.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:15.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:15.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:15.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:15.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:15.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:15.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:15.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:15.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:15.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:15.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:15.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:15.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:15.411+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:15.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:15.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:15.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:15.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:15.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:15.430+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:15.866 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:15.859+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.860919+0000) 2024-02-19T20:24:16.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:15.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.868423+0000) 2024-02-19T20:24:16.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:15.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.960790+0000) 2024-02-19T20:24:16.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:16.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:16.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:16.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:16.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:16.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:16.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:16.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:16.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:16.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:16.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:16.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:16.439 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:16.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:16.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:16.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:16.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:16.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:16.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:16.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:17.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:16.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.868562+0000) 2024-02-19T20:24:17.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:16.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.960942+0000) 2024-02-19T20:24:17.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:16.862+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.863693+0000) 2024-02-19T20:24:17.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:17.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:17.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.028+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:17.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.028+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:17.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:17.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:17.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:17.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:17.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:17.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:17.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:17.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:17.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:17.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:17.725 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:17.463+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:18.188 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:17.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.868711+0000) 2024-02-19T20:24:18.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:17.960+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.961099+0000) 2024-02-19T20:24:18.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.863+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.864410+0000) 2024-02-19T20:24:18.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:18.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:18.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:18.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:17.996+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:18.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:18.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:18.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:18.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:18.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:18.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:18.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:18.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:18.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:18.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:18.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:18.424+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:18.866 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:18.824+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.825544+0000) 2024-02-19T20:24:18.866 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:18.823+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.825265+0000) 2024-02-19T20:24:18.867 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:18.831+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60704 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:18.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:18.827+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.828635+0000) 2024-02-19T20:24:18.867 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:18.863+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.864704+0000) 2024-02-19T20:24:19.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:18.959+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.961260+0000) 2024-02-19T20:24:19.189 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:18.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.868856+0000) 2024-02-19T20:24:19.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:18.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:19.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:18.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:19.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:18.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:19.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:18.992+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:19.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:19.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:19.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:19.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:19.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:19.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:19.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:19.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:19.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:19.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:19.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:19.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:20.170 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:19.867+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.868942+0000) 2024-02-19T20:24:20.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:19.960+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.961423+0000) 2024-02-19T20:24:20.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:19.866+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.867367+0000) 2024-02-19T20:24:20.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:20.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:20.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.024+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:20.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:20.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:20.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:20.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:20.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:20.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:20.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:20.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:20.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:20.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:20.404+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:20.868 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:20.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.869110+0000) 2024-02-19T20:24:21.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:20.960+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.961645+0000) 2024-02-19T20:24:21.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.868+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.870378+0000) 2024-02-19T20:24:21.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:21.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:21.189 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.995+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:21.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:20.995+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:21.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:21.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:21.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:21.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:21.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:21.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:21.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:21.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:21.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:21.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:21.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:21.419+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:21.871 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:21.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.869284+0000) 2024-02-19T20:24:22.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:21.960+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.961804+0000) 2024-02-19T20:24:22.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:21.871+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.873260+0000) 2024-02-19T20:24:22.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:22.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:22.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:22.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:22.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:22.030+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:22.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:22.030+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:22.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:22.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:22.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:22.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:22.725 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:22.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:22.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:22.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:22.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:22.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:22.726 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:22.421+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:23.189 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:22.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.869483+0000) 2024-02-19T20:24:23.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:22.961+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.962033+0000) 2024-02-19T20:24:23.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:22.872+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.874308+0000) 2024-02-19T20:24:23.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.007+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:23.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.007+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:23.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.007+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:23.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.007+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:23.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:23.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:23.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:23.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:23.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:23.376+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:23.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:23.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:23.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:23.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:23.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:23.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:23.872 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:23.840+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60777 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:23.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.836+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.837925+0000) 2024-02-19T20:24:23.873 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:23.833+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.834679+0000) 2024-02-19T20:24:23.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:23.833+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.834834+0000) 2024-02-19T20:24:23.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:23.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.869624+0000) 2024-02-19T20:24:24.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:23.960+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.962226+0000) 2024-02-19T20:24:24.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.873+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.874611+0000) 2024-02-19T20:24:24.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:24.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:24.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.998+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:24.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:23.998+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:24.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:24.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:24.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:24.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:24.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:24.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:24.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:24.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:24.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:24.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:24.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:24.396+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:24.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:24.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.869789+0000) 2024-02-19T20:24:25.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:24.961+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.962380+0000) 2024-02-19T20:24:25.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:24.876+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.877849+0000) 2024-02-19T20:24:25.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:25.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:25.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:25.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.021+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:25.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:25.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:25.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:25.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:25.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:25.392+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:25.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:25.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:25.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:25.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:25.877 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:25.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.869957+0000) 2024-02-19T20:24:26.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:25.961+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.962563+0000) 2024-02-19T20:24:26.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.879+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.880600+0000) 2024-02-19T20:24:26.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:26.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:26.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.979+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:26.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:25.979+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:26.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:26.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:26.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:26.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:26.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:26.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:26.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:26.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:26.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:26.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:26.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:26.358+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:26.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:26.869+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.870132+0000) 2024-02-19T20:24:27.189 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:26.961+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.962709+0000) 2024-02-19T20:24:27.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:26.882+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.883496+0000) 2024-02-19T20:24:27.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:27.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:27.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.000+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:27.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.000+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:27.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:27.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:27.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:27.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:27.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:27.372+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:27.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:27.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:27.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:27.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:27.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:27.325+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:27.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:27.869+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.870302+0000) 2024-02-19T20:24:28.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:27.962+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.962860+0000) 2024-02-19T20:24:28.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.883+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.884694+0000) 2024-02-19T20:24:28.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:28.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:28.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.962+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:28.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:27.962+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:28.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:28.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:28.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:28.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:28.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:28.373+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:28.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:28.276+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:28.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:28.276+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:28.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:28.276+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:28.882 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:28.849+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60851 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:28.883 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:28.845+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.846590+0000) 2024-02-19T20:24:28.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:28.842+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.843513+0000) 2024-02-19T20:24:28.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:28.842+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.843762+0000) 2024-02-19T20:24:28.883 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:28.868+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.870431+0000) 2024-02-19T20:24:29.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:28.961+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.963032+0000) 2024-02-19T20:24:29.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:28.883+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.885282+0000) 2024-02-19T20:24:29.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:28.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:29.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:28.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:29.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:28.946+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:29.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:28.946+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:29.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:29.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:29.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:29.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:29.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:29.316+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:29.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:29.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:29.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:29.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:29.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:29.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:29.885 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:29.869+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.870603+0000) 2024-02-19T20:24:30.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:29.961+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.963126+0000) 2024-02-19T20:24:30.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:29.887+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.888462+0000) 2024-02-19T20:24:30.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:29.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:30.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:29.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:30.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:29.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:30.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:29.919+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:30.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:30.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:30.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:30.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:30.432 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:30.363+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:30.432 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:30.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:30.433 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:30.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:30.433 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:30.282+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:31.190 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:30.870+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.870788+0000) 2024-02-19T20:24:31.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:30.962+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.963301+0000) 2024-02-19T20:24:31.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:30.889+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.891125+0000) 2024-02-19T20:24:31.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:30.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:31.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:30.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:31.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:30.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:31.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:30.917+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:31.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:31.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:31.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:31.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:31.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:31.316+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:31.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:31.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:31.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:31.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:31.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:31.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:31.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:31.870+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.870954+0000) 2024-02-19T20:24:32.190 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:31.962+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.963514+0000) 2024-02-19T20:24:32.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:31.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:32.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:31.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:32.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:31.889+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:32.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:31.889+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:32.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:31.892+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.893728+0000) 2024-02-19T20:24:32.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:32.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:32.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:32.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:32.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:32.278+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:32.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:32.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:32.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:32.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:32.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:32.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:32.891 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:32.869+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.871170+0000) 2024-02-19T20:24:32.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:32.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:32.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:32.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:32.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:32.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:32.892 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:32.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:33.190 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:32.893+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.894381+0000) 2024-02-19T20:24:33.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:32.963+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.963683+0000) 2024-02-19T20:24:33.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:33.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:33.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:33.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:33.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:33.334+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:33.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:33.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:33.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:33.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:33.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:33.310+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:33.891 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:33.857+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60924 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:33.892 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:33.851+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.852429+0000) 2024-02-19T20:24:33.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:33.851+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.853022+0000) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:33.869+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.871311+0000) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:33.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:33.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:33.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:33.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:33.854+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.856100+0000) 2024-02-19T20:24:33.893 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:33.893+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.894653+0000) 2024-02-19T20:24:34.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:33.962+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.963807+0000) 2024-02-19T20:24:34.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:34.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:34.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:34.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:34.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:34.379+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:34.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:34.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:34.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:34.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:34.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:34.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:34.894 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:34.869+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.871401+0000) 2024-02-19T20:24:34.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:34.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:34.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:34.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:34.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:34.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:34.895 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:34.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:35.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:34.896+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.897391+0000) 2024-02-19T20:24:35.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:34.963+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.963972+0000) 2024-02-19T20:24:35.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:35.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:35.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:35.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:35.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:35.373+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:35.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:35.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:35.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:35.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:35.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:35.327+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:36.191 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:35.870+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.871571+0000) 2024-02-19T20:24:36.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:35.963+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.964080+0000) 2024-02-19T20:24:36.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:35.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:36.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:35.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:36.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:35.804+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:36.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:35.804+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:36.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:35.899+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.900406+0000) 2024-02-19T20:24:36.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:36.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:36.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:36.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:36.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:36.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:36.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:36.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:36.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:36.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:36.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:36.374+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:36.900 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:36.871+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.871780+0000) 2024-02-19T20:24:36.900 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:36.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:36.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:36.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:36.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:36.767+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:36.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:36.767+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:37.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:36.902+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.903178+0000) 2024-02-19T20:24:37.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:36.963+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.964277+0000) 2024-02-19T20:24:37.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:37.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:37.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:37.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:37.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:37.333+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:37.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:37.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:37.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:37.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:37.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:37.332+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:37.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:37.870+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.871880+0000) 2024-02-19T20:24:37.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:37.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:37.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:37.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:37.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:37.733+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:37.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:37.733+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:38.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:37.902+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.904095+0000) 2024-02-19T20:24:38.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:37.963+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.964461+0000) 2024-02-19T20:24:38.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:38.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:38.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:38.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:38.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:38.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:38.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:38.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:38.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:38.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:38.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:38.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:38.885 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:38.866+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 60998 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:38.885 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:38.860+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.861704+0000) 2024-02-19T20:24:38.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:38.859+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.860814+0000) 2024-02-19T20:24:38.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:38.870+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.871994+0000) 2024-02-19T20:24:38.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:38.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:38.886 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:38.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:38.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:38.732+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:38.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:38.732+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:38.887 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:38.863+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.864527+0000) 2024-02-19T20:24:39.191 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:38.902+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.904354+0000) 2024-02-19T20:24:39.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:38.964+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.964621+0000) 2024-02-19T20:24:39.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:39.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:39.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:39.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:39.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:39.415+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:39.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:39.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:39.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:39.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:39.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:39.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:39.904 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:39.871+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.872114+0000) 2024-02-19T20:24:39.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:39.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:39.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:39.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:39.906 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:39.725+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:39.907 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:39.725+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:39.907 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:39.906+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.906854+0000) 2024-02-19T20:24:40.170 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:39.964+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.964768+0000) 2024-02-19T20:24:40.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:40.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:40.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:40.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:40.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:40.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:40.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:40.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:40.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:40.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:40.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:40.388+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:40.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:40.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:40.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:40.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:40.869 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:40.751+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:40.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:40.751+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:41.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:40.871+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.872281+0000) 2024-02-19T20:24:41.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:40.909+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.910051+0000) 2024-02-19T20:24:41.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:40.964+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.964941+0000) 2024-02-19T20:24:41.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:41.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:41.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:41.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:41.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:41.300+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:41.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:41.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:41.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:41.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:41.476 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:41.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:41.910 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:41.871+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.872469+0000) 2024-02-19T20:24:41.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:41.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:41.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:41.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:41.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:41.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:41.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:41.753+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:41.911 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:41.913+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.913749+0000) 2024-02-19T20:24:42.191 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:41.964+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.965151+0000) 2024-02-19T20:24:42.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:42.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:42.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:42.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:42.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:42.358+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:42.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:42.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:42.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:42.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:42.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:42.283+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:42.911 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:42.872+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.872642+0000) 2024-02-19T20:24:42.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:42.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:42.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:42.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:42.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:42.783+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:42.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:42.783+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:42.912 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:42.914+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.914853+0000) 2024-02-19T20:24:43.192 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:42.964+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.965300+0000) 2024-02-19T20:24:43.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:43.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:43.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:43.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:43.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:43.331+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:43.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:43.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:43.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:43.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:43.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:43.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:43.895 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:43.875+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61071 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:43.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:43.869+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.869647+0000) 2024-02-19T20:24:43.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:43.868+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.869029+0000) 2024-02-19T20:24:43.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:43.872+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.872753+0000) 2024-02-19T20:24:43.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:43.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:43.896 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:43.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:43.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:43.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:43.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:43.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:43.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:43.872+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.872596+0000) 2024-02-19T20:24:44.186 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:43.914+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.915121+0000) 2024-02-19T20:24:44.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:43.965+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.965475+0000) 2024-02-19T20:24:44.187 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:44.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:44.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:44.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:44.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:44.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:44.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:44.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:44.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:44.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:44.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:44.359+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:44.915 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:44.872+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.872918+0000) 2024-02-19T20:24:44.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:44.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:44.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:44.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:44.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:44.796+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:44.916 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:44.796+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:45.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:44.917+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.917725+0000) 2024-02-19T20:24:45.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:44.965+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.965594+0000) 2024-02-19T20:24:45.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:45.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:45.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:45.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:45.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:45.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:45.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:45.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:45.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:45.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:45.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:45.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:45.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:45.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:45.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:45.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:45.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:45.793+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:45.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:45.794+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:46.192 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:45.872+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.873049+0000) 2024-02-19T20:24:46.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:45.920+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.920526+0000) 2024-02-19T20:24:46.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:45.965+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.965759+0000) 2024-02-19T20:24:46.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:46.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:46.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:46.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:46.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:46.247+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:46.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:46.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:46.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:46.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:46.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:46.344+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:46.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:46.872+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.873172+0000) 2024-02-19T20:24:46.920 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:46.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:46.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:46.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:46.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:46.842+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:46.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:46.842+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:46.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:46.922+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.923253+0000) 2024-02-19T20:24:47.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:46.965+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.965924+0000) 2024-02-19T20:24:47.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:47.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:47.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:47.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:47.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:47.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:47.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:47.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:47.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:47.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:47.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:47.379+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:47.921 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:47.873+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.873374+0000) 2024-02-19T20:24:47.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:47.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:47.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:47.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:47.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:47.840+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:47.922 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:47.840+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:48.192 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:47.923+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.924329+0000) 2024-02-19T20:24:48.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:47.965+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.966020+0000) 2024-02-19T20:24:48.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:48.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:48.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:48.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:48.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:48.305+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:48.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:48.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:48.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:48.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:48.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:48.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:48.899 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:48.884+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61145 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:48.900 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:48.878+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.879148+0000) 2024-02-19T20:24:48.900 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:48.873+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.873635+0000) 2024-02-19T20:24:48.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:48.877+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.878093+0000) 2024-02-19T20:24:48.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:48.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:48.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:48.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:48.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:48.791+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:48.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:48.791+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:48.901 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:48.881+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.881499+0000) 2024-02-19T20:24:49.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:48.924+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.924648+0000) 2024-02-19T20:24:49.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:48.965+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.966147+0000) 2024-02-19T20:24:49.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:49.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:49.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:49.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:49.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:49.290+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:49.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:49.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:49.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:49.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:49.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:49.389+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:49.924 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:49.873+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.873821+0000) 2024-02-19T20:24:49.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:49.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:49.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:49.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:49.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:49.753+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:49.925 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:49.753+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:50.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:49.926+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.926975+0000) 2024-02-19T20:24:50.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:49.966+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.966338+0000) 2024-02-19T20:24:50.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:50.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:50.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:50.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:50.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:50.356+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:50.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:50.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:50.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:50.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:50.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:50.280+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:50.870 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:50.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:50.871 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:50.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:50.871 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:50.704+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:50.871 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:50.704+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:51.193 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:50.873+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.873913+0000) 2024-02-19T20:24:51.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:50.929+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.930188+0000) 2024-02-19T20:24:51.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:50.966+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.966547+0000) 2024-02-19T20:24:51.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:51.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:51.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:51.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:51.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:51.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:51.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:51.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:51.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:51.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:51.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:51.269+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:51.930 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:51.873+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.874052+0000) 2024-02-19T20:24:51.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:51.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:51.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:51.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:51.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:51.665+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:51.930 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:51.665+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:52.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:51.932+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.933017+0000) 2024-02-19T20:24:52.193 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:51.966+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.966694+0000) 2024-02-19T20:24:52.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:52.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:52.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:52.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:52.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:52.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:52.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:52.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:52.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:52.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:52.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:52.281+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:52.931 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:52.873+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.874243+0000) 2024-02-19T20:24:52.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:52.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:52.931 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:52.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:52.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:52.705+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:52.932 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:52.705+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:53.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:52.933+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.934242+0000) 2024-02-19T20:24:53.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:52.966+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.966849+0000) 2024-02-19T20:24:53.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:53.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:53.474 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:53.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:53.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:53.378+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:53.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:53.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:53.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:53.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:53.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:53.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:53.900 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:53.893+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61218 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:53.901 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:53.886+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.887234+0000) 2024-02-19T20:24:53.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:53.874+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.874433+0000) 2024-02-19T20:24:53.901 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:53.886+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.887002+0000) 2024-02-19T20:24:53.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:53.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:53.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:53.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:53.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:53.709+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:53.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:53.709+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:53.902 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:53.891+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.891694+0000) 2024-02-19T20:24:54.193 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:53.934+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.934569+0000) 2024-02-19T20:24:54.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:53.966+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.966971+0000) 2024-02-19T20:24:54.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:54.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:54.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:54.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:54.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:54.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:54.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:54.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:54.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:54.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:54.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:54.407+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:54.933 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:54.874+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.874643+0000) 2024-02-19T20:24:54.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:54.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:54.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:54.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:54.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:54.702+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:54.934 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:54.702+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:55.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:54.936+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.937043+0000) 2024-02-19T20:24:55.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:54.966+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.967119+0000) 2024-02-19T20:24:55.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:55.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:55.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:55.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:55.475 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:55.204+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:55.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:55.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:55.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:55.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:55.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:55.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:55.920 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:55.874+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.874750+0000) 2024-02-19T20:24:55.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:55.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:55.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:55.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:55.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:55.698+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:55.921 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:55.698+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:56.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:55.939+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.939697+0000) 2024-02-19T20:24:56.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:55.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.967299+0000) 2024-02-19T20:24:56.180 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:56.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:56.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:56.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:56.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:56.182+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:56.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:56.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:56.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:56.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:56.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:56.424+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:56.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:56.874+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.874879+0000) 2024-02-19T20:24:56.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:56.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:56.939 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:56.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:56.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:56.690+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:56.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:56.690+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:56.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:56.942+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.942569+0000) 2024-02-19T20:24:57.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:56.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.967506+0000) 2024-02-19T20:24:57.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:57.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:57.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:57.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:57.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:57.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:57.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:57.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:57.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:57.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:57.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:57.443+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:57.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:57.874+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.875005+0000) 2024-02-19T20:24:57.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:57.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:57.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:57.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:57.940 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:57.689+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:57.941 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:57.689+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:58.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:57.943+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.943520+0000) 2024-02-19T20:24:58.194 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:57.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.967731+0000) 2024-02-19T20:24:58.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:58.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:58.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:58.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:58.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:58.212+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:58.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:58.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:58.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:58.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:58.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:58.480+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:58.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:58.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:58.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:58.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:58.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:58.648+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:58.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:58.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:59.163 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:24:58.903+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61292 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:24:59.164 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:58.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.875183+0000) 2024-02-19T20:24:59.164 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:58.895+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.895733+0000) 2024-02-19T20:24:59.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:58.898+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.899375+0000) 2024-02-19T20:24:59.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:58.943+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.943773+0000) 2024-02-19T20:24:59.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:58.895+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.895914+0000) 2024-02-19T20:24:59.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:58.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.967896+0000) 2024-02-19T20:24:59.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:59.166+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:59.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:59.166+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:59.165 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:59.166+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:24:59.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:59.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:59.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:59.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:59.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:59.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:24:59.872 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:59.657+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:59.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:59.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:59.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:59.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:24:59.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:59.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:00.122 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:24:59.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.875386+0000) 2024-02-19T20:25:00.122 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:24:59.946+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.946569+0000) 2024-02-19T20:25:00.122 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:24:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:24:59.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.968067+0000) 2024-02-19T20:25:00.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:00.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:00.123 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:00.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:00.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:00.125+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:00.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:00.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:00.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:00.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:00.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:00.533+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:00.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:00.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:00.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:00.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:00.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:00.650+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:00.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:00.650+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:01.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:00.949+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.949560+0000) 2024-02-19T20:25:01.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:00.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.875570+0000) 2024-02-19T20:25:01.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:00.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.968189+0000) 2024-02-19T20:25:01.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:01.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:01.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:01.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:01.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:01.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:01.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:01.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:01.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:01.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:01.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:01.555+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:01.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:01.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:01.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:01.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:01.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:01.614+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:01.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:01.614+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:02.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:01.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.875801+0000) 2024-02-19T20:25:02.195 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:01.952+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.952491+0000) 2024-02-19T20:25:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:01.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.968378+0000) 2024-02-19T20:25:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:02.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:02.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:02.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:02.087+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:02.872 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:02.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:02.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:02.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:02.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:02.582+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:02.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:02.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:02.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:02.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:02.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:02.655+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:02.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:02.655+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:03.128 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:02.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.875945+0000) 2024-02-19T20:25:03.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:02.952+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.953308+0000) 2024-02-19T20:25:03.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:02.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.968552+0000) 2024-02-19T20:25:03.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:03.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:03.129 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:03.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:03.130 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:03.132+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:03.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:03.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:03.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:03.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:03.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:03.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:03.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:03.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:03.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:03.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:03.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:03.683+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:03.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:03.683+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:04.171 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:03.912+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61364 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:04.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:03.907+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.908115+0000) 2024-02-19T20:25:04.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:03.953+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.953578+0000) 2024-02-19T20:25:04.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:03.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.876074+0000) 2024-02-19T20:25:04.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:03.905+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.905513+0000) 2024-02-19T20:25:04.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:03.905+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.906078+0000) 2024-02-19T20:25:04.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:03.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.968692+0000) 2024-02-19T20:25:04.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:04.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:04.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:04.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:04.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:04.174+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:04.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:04.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:04.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:04.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:04.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:04.656+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:04.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:04.656+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:04.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:04.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:04.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:04.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:04.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:04.561+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:04.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:04.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.876255+0000) 2024-02-19T20:25:05.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:04.955+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.956097+0000) 2024-02-19T20:25:05.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:04.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.968822+0000) 2024-02-19T20:25:05.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:05.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:05.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:05.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:05.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:05.200+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:05.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:05.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:05.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:05.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:05.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:05.590+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:05.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:05.874+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.876382+0000) 2024-02-19T20:25:05.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:05.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:05.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:05.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:05.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:05.613+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:05.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:05.613+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:06.176 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:05.957+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.958925+0000) 2024-02-19T20:25:06.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:05.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.969001+0000) 2024-02-19T20:25:06.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:06.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:06.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:06.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:06.177 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:06.178+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:06.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:06.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:06.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:06.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:06.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:06.545+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:06.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:06.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:06.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:06.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:06.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:06.619+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:06.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:06.619+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:07.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:06.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.876554+0000) 2024-02-19T20:25:07.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:06.960+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.961667+0000) 2024-02-19T20:25:07.196 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:06.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.969196+0000) 2024-02-19T20:25:07.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:07.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:07.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:07.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:07.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:07.205+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:07.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:07.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:07.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:07.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:07.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:07.550+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:07.873 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:07.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:07.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:07.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:07.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:07.629+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:07.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:07.629+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:08.171 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:07.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.876684+0000) 2024-02-19T20:25:08.171 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:07.961+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.962511+0000) 2024-02-19T20:25:08.172 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:07.967+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.969355+0000) 2024-02-19T20:25:08.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:08.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:08.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:08.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:08.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:08.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:08.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:08.528+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:08.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:08.528+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:08.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:08.528+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:08.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:08.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:08.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:08.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:08.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:08.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:08.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:08.594+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:09.150 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:08.920+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61439 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:09.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:08.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.876868+0000) 2024-02-19T20:25:09.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:08.913+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.914822+0000) 2024-02-19T20:25:09.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:08.917+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.918757+0000) 2024-02-19T20:25:09.151 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:08.961+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.962818+0000) 2024-02-19T20:25:09.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:08.913+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.914900+0000) 2024-02-19T20:25:09.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:08.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.969524+0000) 2024-02-19T20:25:09.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:09.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:09.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:09.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:09.152 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:09.152+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:09.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:09.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:09.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:09.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:09.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:09.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:09.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:09.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:09.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:09.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:09.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:09.592+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:09.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:09.592+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:10.135 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:09.876+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.877099+0000) 2024-02-19T20:25:10.136 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:09.964+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.965430+0000) 2024-02-19T20:25:10.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:09.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.969626+0000) 2024-02-19T20:25:10.136 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:10.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:10.137 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:10.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:10.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:10.138+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:10.764 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:10.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:10.764 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:10.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:10.765 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:10.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:10.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:10.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:10.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:10.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:10.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:10.547+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:10.765 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:10.547+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:11.038 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:10.875+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.877211+0000) 2024-02-19T20:25:11.038 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:10.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.969732+0000) 2024-02-19T20:25:11.038 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:10.966+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.968066+0000) 2024-02-19T20:25:11.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:11.088+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:11.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:11.088+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:11.474 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:11.088+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:11.873 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:11.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:11.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:11.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:11.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:11.546+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:11.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:11.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:11.874 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:11.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:11.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:11.568+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:11.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:11.568+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:12.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:11.876+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.877430+0000) 2024-02-19T20:25:12.196 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:11.969+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.970737+0000) 2024-02-19T20:25:12.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:11.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.969890+0000) 2024-02-19T20:25:12.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:12.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:12.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:12.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:12.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:12.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:12.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:12.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:12.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:12.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:12.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:12.522+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:12.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:12.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:12.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:12.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:13.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:12.876+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.877577+0000) 2024-02-19T20:25:13.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:12.970+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.971510+0000) 2024-02-19T20:25:13.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:12.969+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.970052+0000) 2024-02-19T20:25:13.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:13.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:13.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:13.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:13.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:13.036+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:13.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:13.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:13.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:13.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:13.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:13.560+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:13.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:13.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:13.838 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:13.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:13.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:13.489+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:13.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:13.490+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:14.196 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:13.929+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61511 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:14.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:13.876+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.877747+0000) 2024-02-19T20:25:14.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:13.922+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.924024+0000) 2024-02-19T20:25:14.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:13.925+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.926963+0000) 2024-02-19T20:25:14.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:13.970+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.971738+0000) 2024-02-19T20:25:14.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:13.922+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.924285+0000) 2024-02-19T20:25:14.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:13.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.970243+0000) 2024-02-19T20:25:14.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:14.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:14.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:14.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:14.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:14.060+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:14.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:14.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:14.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:14.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:14.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:14.597+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:14.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:14.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:14.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:14.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:14.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:14.468+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:14.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:14.469+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:15.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:14.876+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.877864+0000) 2024-02-19T20:25:15.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:14.972+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.974259+0000) 2024-02-19T20:25:15.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:14.968+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.970421+0000) 2024-02-19T20:25:15.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:15.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:15.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:15.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:15.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:15.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:15.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:15.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:15.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:15.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:15.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:15.626+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:15.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:15.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:15.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:15.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:15.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:15.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:15.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:15.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:16.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:15.876+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.877963+0000) 2024-02-19T20:25:16.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:15.975+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.976977+0000) 2024-02-19T20:25:16.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:15.969+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.970650+0000) 2024-02-19T20:25:16.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:16.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:16.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.002+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:16.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:16.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:16.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:16.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:16.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:16.600+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:16.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:16.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:16.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:16.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:16.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:16.520+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:16.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:16.520+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:17.196 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:16.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.878146+0000) 2024-02-19T20:25:17.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:16.978+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.979671+0000) 2024-02-19T20:25:17.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:17.197 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:17.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.965+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:17.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:16.969+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.970805+0000) 2024-02-19T20:25:17.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:17.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:17.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:17.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:17.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:17.603+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:17.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:17.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:17.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:17.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:17.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:17.565+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:17.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:17.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:18.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:17.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.878347+0000) 2024-02-19T20:25:18.197 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:17.979+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.980654+0000) 2024-02-19T20:25:18.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:17.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.971006+0000) 2024-02-19T20:25:18.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:17.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:18.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:17.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:18.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:17.981+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:18.874 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:18.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:18.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:18.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:18.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:18.631+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:18.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:18.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:18.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:18.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:18.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:18.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:18.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:18.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:19.197 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:18.938+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61586 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:19.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:18.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.878534+0000) 2024-02-19T20:25:19.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:18.931+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.933119+0000) 2024-02-19T20:25:19.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:18.934+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.935956+0000) 2024-02-19T20:25:19.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:18.979+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.980939+0000) 2024-02-19T20:25:19.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:18.931+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.932770+0000) 2024-02-19T20:25:19.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:18.969+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.971126+0000) 2024-02-19T20:25:19.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:19.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:19.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:19.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:19.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:19.007+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:19.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:19.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:19.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:19.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:19.875 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:19.661+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:19.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:19.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:19.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:19.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:19.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:19.494+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:19.876 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:19.494+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:20.172 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:19.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.878700+0000) 2024-02-19T20:25:20.172 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:19.982+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.983830+0000) 2024-02-19T20:25:20.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:19.969+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.971267+0000) 2024-02-19T20:25:20.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:20.045+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:20.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:20.045+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:20.173 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:20.045+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:20.723 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:20.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:20.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:20.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:20.724 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:20.634+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:20.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:20.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:20.724 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:20.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:20.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:20.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:20.725 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:20.449+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:21.197 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:20.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.878829+0000) 2024-02-19T20:25:21.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:20.984+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.986030+0000) 2024-02-19T20:25:21.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:20.969+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.971376+0000) 2024-02-19T20:25:21.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:21.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:21.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:21.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:21.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:21.075+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:21.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:21.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:21.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:21.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:21.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:21.448+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:21.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:21.448+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:21.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:21.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:21.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:21.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:21.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:21.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:21.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:21.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.878943+0000) 2024-02-19T20:25:22.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:21.987+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.989250+0000) 2024-02-19T20:25:22.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:21.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.971527+0000) 2024-02-19T20:25:22.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:22.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:22.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:22.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:22.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:22.101+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:22.670 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:22.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:22.670 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:22.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:22.670 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:22.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:22.671 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:22.474+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:22.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:22.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:22.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:22.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:22.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:22.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:22.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:22.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.879119+0000) 2024-02-19T20:25:23.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:22.988+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.989828+0000) 2024-02-19T20:25:23.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:22.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.971620+0000) 2024-02-19T20:25:23.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:23.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:23.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:23.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:23.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:23.091+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:23.670 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:23.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:23.670 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:23.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:23.670 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:23.454+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:23.671 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:23.454+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:23.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:23.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:23.939 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:23.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:23.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:23.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:23.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:23.877+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.879293+0000) 2024-02-19T20:25:23.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:23.939+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.940753+0000) 2024-02-19T20:25:24.197 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:23.945+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61658 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:24.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:23.942+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.944138+0000) 2024-02-19T20:25:24.198 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:23.988+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.989984+0000) 2024-02-19T20:25:24.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:23.939+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.940999+0000) 2024-02-19T20:25:24.198 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:23.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.971732+0000) 2024-02-19T20:25:24.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:24.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:24.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:24.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:24.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:24.089+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:24.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:24.405+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:24.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:24.405+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:24.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:24.405+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:24.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:24.405+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:24.967 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:24.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:24.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:24.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:24.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:24.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:24.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:24.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.879479+0000) 2024-02-19T20:25:25.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:24.990+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.992003+0000) 2024-02-19T20:25:25.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:24.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.971839+0000) 2024-02-19T20:25:25.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:25.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:25.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:25.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:25.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:25.094+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:25.672 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:25.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:25.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:25.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:25.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:25.396+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:25.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:25.396+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:25.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:25.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:25.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:25.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:25.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:25.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:25.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:25.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.879581+0000) 2024-02-19T20:25:26.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:25.992+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.994109+0000) 2024-02-19T20:25:26.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:25.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.971944+0000) 2024-02-19T20:25:26.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:26.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:26.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:26.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:26.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:26.106+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:26.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:26.381+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:26.639 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:26.381+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:26.640 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:26.381+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:26.640 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:26.381+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:26.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:26.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:26.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:26.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:26.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:26.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:26.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:26.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.879694+0000) 2024-02-19T20:25:27.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:26.994+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.996333+0000) 2024-02-19T20:25:27.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:26.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.972061+0000) 2024-02-19T20:25:27.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:27.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:27.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:27.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:27.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:27.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:27.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:27.383+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:27.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:27.383+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:27.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:27.383+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:27.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:27.384+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:27.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:27.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:27.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:27.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:27.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:27.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:27.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:27.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.879811+0000) 2024-02-19T20:25:28.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:27.995+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.996923+0000) 2024-02-19T20:25:28.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:27.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.972171+0000) 2024-02-19T20:25:28.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:28.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:28.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:28.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:28.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:28.159+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:28.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:28.348+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:28.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:28.348+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:28.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:28.348+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:28.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:28.348+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:28.943 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:28.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:28.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:28.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:28.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:28.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:28.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:28.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.879951+0000) 2024-02-19T20:25:28.944 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:28.946+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.947674+0000) 2024-02-19T20:25:29.198 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:28.952+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61733 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:29.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:28.949+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.950701+0000) 2024-02-19T20:25:29.199 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:28.995+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.997160+0000) 2024-02-19T20:25:29.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:28.946+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.948434+0000) 2024-02-19T20:25:29.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:28.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.972278+0000) 2024-02-19T20:25:29.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:29.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:29.199 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:29.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:29.200 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:29.144+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:29.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:29.343+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:29.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:29.343+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:29.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:29.343+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:29.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:29.343+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:29.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:29.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:29.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:29.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:29.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:29.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:29.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:29.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.880093+0000) 2024-02-19T20:25:30.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:29.997+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.999207+0000) 2024-02-19T20:25:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:29.970+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.972410+0000) 2024-02-19T20:25:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:30.124+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:30.124+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:30.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:30.124+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:30.676 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:30.377+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:30.676 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:30.377+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:30.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:30.377+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:30.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:30.377+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:30.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:30.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:30.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:30.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:30.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:30.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:30.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:30.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.880229+0000) 2024-02-19T20:25:31.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:30.999+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.001308+0000) 2024-02-19T20:25:31.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:30.971+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.972554+0000) 2024-02-19T20:25:31.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:31.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:31.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:31.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:31.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:31.100+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:31.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:31.384+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:31.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:31.384+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:31.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:31.384+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:31.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:31.384+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:31.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:31.727+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:31.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:31.727+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:31.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:31.727+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:31.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:31.878+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.880325+0000) 2024-02-19T20:25:32.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:32.002+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.003432+0000) 2024-02-19T20:25:32.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:31.971+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.972669+0000) 2024-02-19T20:25:32.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:32.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:32.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:32.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:32.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:32.118+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:32.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:32.385+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:32.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:32.385+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:32.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:32.385+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:32.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:32.385+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:32.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:32.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:32.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:32.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:32.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:32.712+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:32.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:32.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.880479+0000) 2024-02-19T20:25:33.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:33.003+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.004207+0000) 2024-02-19T20:25:33.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:32.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.972758+0000) 2024-02-19T20:25:33.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:33.156+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:33.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:33.156+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:33.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:33.156+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:33.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:33.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:33.668 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:33.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:33.668 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:33.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:33.668 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:33.404+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:33.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:33.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:33.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:33.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:33.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:33.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:33.951 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:33.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.880631+0000) 2024-02-19T20:25:34.223 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:33.954+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.955375+0000) 2024-02-19T20:25:34.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:33.959+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.959624+0000) 2024-02-19T20:25:34.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:34.003+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.004400+0000) 2024-02-19T20:25:34.224 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:33.959+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61805 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:34.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:33.954+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.955276+0000) 2024-02-19T20:25:34.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:33.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.972859+0000) 2024-02-19T20:25:34.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:34.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:34.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:34.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:34.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:34.154+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:34.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:34.358+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:34.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:34.358+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:34.677 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:34.358+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:34.678 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:34.358+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:34.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:34.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:34.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:34.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:34.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:34.711+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:34.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:34.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.880721+0000) 2024-02-19T20:25:35.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:35.006+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.006456+0000) 2024-02-19T20:25:35.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:34.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.973013+0000) 2024-02-19T20:25:35.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:35.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:35.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:35.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:35.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:35.199+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:35.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:35.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:35.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:35.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:35.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:35.401+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:35.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:35.401+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:35.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:35.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:35.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:35.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:35.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:35.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:35.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:35.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.880817+0000) 2024-02-19T20:25:36.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:36.009+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.009963+0000) 2024-02-19T20:25:36.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:35.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.973129+0000) 2024-02-19T20:25:36.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:36.203+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:36.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:36.203+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:36.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:36.203+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:36.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:36.426+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:36.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:36.426+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:36.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:36.426+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:36.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:36.426+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:36.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:36.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:36.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:36.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:36.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:36.689+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:36.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:36.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.880912+0000) 2024-02-19T20:25:37.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:37.013+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.013455+0000) 2024-02-19T20:25:37.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:36.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.973235+0000) 2024-02-19T20:25:37.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:37.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:37.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:37.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:37.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:37.180+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:37.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:37.389+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:37.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:37.389+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:37.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:37.389+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:37.674 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:37.389+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:37.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:37.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:37.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:37.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:37.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:37.677+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:37.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:37.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.881030+0000) 2024-02-19T20:25:38.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:38.013+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.014404+0000) 2024-02-19T20:25:38.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:37.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.973329+0000) 2024-02-19T20:25:38.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:38.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:38.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:38.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:38.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:38.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:38.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:38.409+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:38.680 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:38.409+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:38.681 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:38.409+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:38.681 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:38.409+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:38.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:38.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:38.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:38.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:38.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:38.719+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:38.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:38.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.881122+0000) 2024-02-19T20:25:38.958 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:38.961+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.961640+0000) 2024-02-19T20:25:39.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:38.966+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61880 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:39.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:38.965+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.965619+0000) 2024-02-19T20:25:39.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:39.014+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.014587+0000) 2024-02-19T20:25:39.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:38.961+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.962045+0000) 2024-02-19T20:25:39.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:38.972+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.973424+0000) 2024-02-19T20:25:39.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:39.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:39.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:39.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:39.225 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:39.191+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:39.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:39.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:39.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:39.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:39.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:39.451+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:39.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:39.451+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:39.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:39.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:39.754+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:39.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:39.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.881274+0000) 2024-02-19T20:25:40.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:40.016+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.016619+0000) 2024-02-19T20:25:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:39.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.973514+0000) 2024-02-19T20:25:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:40.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:40.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:40.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:40.186+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:40.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:40.403+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:40.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:40.403+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:40.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:40.403+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:40.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:40.403+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:40.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:40.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:40.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:40.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:40.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:40.723+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:40.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:40.880+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.881384+0000) 2024-02-19T20:25:41.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:41.018+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.018652+0000) 2024-02-19T20:25:41.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:40.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.973662+0000) 2024-02-19T20:25:41.675 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:41.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:41.675 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:41.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:41.675 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:41.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:41.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:41.440+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:41.676 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:41.440+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:41.676 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:41.440+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:41.676 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:41.440+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:41.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:41.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:41.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:41.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:41.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:41.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:41.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:41.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.881508+0000) 2024-02-19T20:25:42.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:41.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.973756+0000) 2024-02-19T20:25:42.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:42.020+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.020755+0000) 2024-02-19T20:25:42.683 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:42.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:42.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:42.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:42.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:42.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:42.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:42.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:42.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:42.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:42.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:42.430+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:42.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:42.430+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:42.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:42.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:42.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:42.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:42.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:42.688+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:42.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:42.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.881594+0000) 2024-02-19T20:25:43.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:43.021+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.021437+0000) 2024-02-19T20:25:43.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:42.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.973857+0000) 2024-02-19T20:25:43.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:43.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:43.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:43.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:43.655 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:43.240+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:43.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:43.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:43.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:43.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:43.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:43.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:43.656 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:43.459+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:43.963 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:43.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:43.964 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:43.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:43.964 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:43.658+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:43.964 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:43.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.881701+0000) 2024-02-19T20:25:44.223 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:43.968+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.968750+0000) 2024-02-19T20:25:44.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:43.973+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 61952 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:44.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:43.967+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.968320+0000) 2024-02-19T20:25:44.224 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:43.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.973979+0000) 2024-02-19T20:25:44.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:43.971+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.972090+0000) 2024-02-19T20:25:44.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:44.021+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.021713+0000) 2024-02-19T20:25:44.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:44.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:44.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:44.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:44.645 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:44.234+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:44.645 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:44.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:44.645 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:44.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:44.646 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:44.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:44.646 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:44.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:44.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:44.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:44.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:44.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:44.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:44.649+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:44.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:44.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.881846+0000) 2024-02-19T20:25:45.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:45.023+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.023744+0000) 2024-02-19T20:25:45.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:44.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.974078+0000) 2024-02-19T20:25:45.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:45.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:45.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:45.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:45.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:45.198+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:45.663 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:45.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:45.663 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:45.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:45.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:45.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:45.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:45.497+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:45.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:45.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:45.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:45.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:45.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:45.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:45.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:45.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.881963+0000) 2024-02-19T20:25:46.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:46.025+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.026012+0000) 2024-02-19T20:25:46.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:45.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.974189+0000) 2024-02-19T20:25:46.653 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:46.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:46.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:46.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:46.654 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:46.231+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:46.654 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:46.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:46.654 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:46.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:46.654 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:46.477+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:46.655 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:46.478+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:46.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:46.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:46.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:46.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:46.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:46.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:46.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:46.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.882049+0000) 2024-02-19T20:25:47.221 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:47.027+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.028052+0000) 2024-02-19T20:25:47.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:46.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.974296+0000) 2024-02-19T20:25:47.221 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:47.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:47.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:47.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:47.222 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:47.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:47.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:47.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:47.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:47.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:47.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:47.447+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:47.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:47.447+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:47.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:47.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:47.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:47.676+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:47.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.882137+0000) 2024-02-19T20:25:48.223 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:48.028+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.029058+0000) 2024-02-19T20:25:48.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:47.973+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.974407+0000) 2024-02-19T20:25:48.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:48.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:48.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:48.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:48.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:48.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:48.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:48.431+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:48.688 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:48.431+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:48.688 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:48.431+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:48.688 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:48.431+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:48.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:48.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:48.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:48.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:48.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:48.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:48.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:48.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.882251+0000) 2024-02-19T20:25:49.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:48.979+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62027 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:49.223 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:48.974+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.974683+0000) 2024-02-19T20:25:49.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:48.974+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.975354+0000) 2024-02-19T20:25:49.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:48.978+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.978763+0000) 2024-02-19T20:25:49.224 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:49.028+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.029251+0000) 2024-02-19T20:25:49.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:49.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:49.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:49.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:49.664 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:49.251+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:49.664 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:49.472+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:49.665 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:49.472+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:49.665 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:49.472+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:49.665 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:49.472+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:49.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:49.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:49.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:49.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:49.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:49.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:49.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:49.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.882407+0000) 2024-02-19T20:25:50.286 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:50.030+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.031296+0000) 2024-02-19T20:25:50.286 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:50.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:50.286 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:50.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:50.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:50.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:50.661 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:50.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:50.662 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:50.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:50.662 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:50.459+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:50.662 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:50.459+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:50.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:50.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:50.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:50.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:50.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:50.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:50.974 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:50.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.882601+0000) 2024-02-19T20:25:51.286 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:51.032+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.033351+0000) 2024-02-19T20:25:51.286 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:51.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:51.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:51.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:51.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:51.229+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:51.620 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:51.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:51.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:51.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:51.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:51.449+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:51.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:51.449+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:51.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:51.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:51.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:51.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:51.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:51.624+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:52.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:51.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.882734+0000) 2024-02-19T20:25:52.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:52.036+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.036813+0000) 2024-02-19T20:25:52.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:52.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:52.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:52.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:52.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:52.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:52.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:52.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:52.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:52.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:52.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:52.442+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:52.474 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:52.442+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:52.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:52.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:52.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:52.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:52.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:52.581+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:52.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:52.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.882863+0000) 2024-02-19T20:25:53.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:53.037+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.037901+0000) 2024-02-19T20:25:53.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:53.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:53.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:53.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:53.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:53.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:53.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:53.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:53.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:53.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:53.839 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:53.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:53.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:53.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:53.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:53.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:53.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:53.488+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:53.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:53.488+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:54.101 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:53.987+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62099 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:54.102 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:53.981+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.981651+0000) 2024-02-19T20:25:54.102 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:53.984+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.985127+0000) 2024-02-19T20:25:54.103 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:54.037+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.038181+0000) 2024-02-19T20:25:54.103 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:53.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.882957+0000) 2024-02-19T20:25:54.103 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:53.980+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.981368+0000) 2024-02-19T20:25:54.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:54.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:54.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:54.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:54.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:54.225+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:54.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:54.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:54.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:54.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:54.878 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:54.500+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:54.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:54.526+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:54.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:54.526+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:54.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:54.526+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:54.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:54.526+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:55.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:54.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.883070+0000) 2024-02-19T20:25:55.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:55.041+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.041666+0000) 2024-02-19T20:25:55.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:55.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:55.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:55.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:55.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:55.227+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:55.878 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:55.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.724669+0000) 2024-02-19T20:25:55.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:55.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:55.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:55.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:55.879 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:55.535+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:55.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:55.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:55.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:55.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:55.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:55.497+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:55.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:55.497+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:56.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:56.043+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.045111+0000) 2024-02-19T20:25:56.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:55.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.883164+0000) 2024-02-19T20:25:56.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:56.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:56.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:56.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:56.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:56.248+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:56.807 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:56.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.724768+0000) 2024-02-19T20:25:56.808 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:56.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:56.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:56.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:56.809 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:56.548+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:56.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:56.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:56.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:56.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:56.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:56.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:56.810 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:56.533+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:57.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:56.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.883271+0000) 2024-02-19T20:25:57.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:57.045+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.047183+0000) 2024-02-19T20:25:57.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:57.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:57.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:57.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:57.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:57.293+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:57.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:57.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.724884+0000) 2024-02-19T20:25:57.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:57.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:57.880 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:57.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:57.881 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:57.530+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:57.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:57.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:57.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:57.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:57.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:57.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:57.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:57.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:58.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:57.881+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.883377+0000) 2024-02-19T20:25:58.201 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:58.046+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.047906+0000) 2024-02-19T20:25:58.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:58.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:58.649 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:58.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:58.649 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:58.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:58.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:58.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:58.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:58.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:58.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:58.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:58.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:58.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:58.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:58.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:58.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:58.473+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:58.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:58.473+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:58.973 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:58.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.724994+0000) 2024-02-19T20:25:58.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:58.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.883480+0000) 2024-02-19T20:25:59.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:58.988+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.989493+0000) 2024-02-19T20:25:59.288 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:25:58.992+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62174 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:25:59.288 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:58.987+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.988700+0000) 2024-02-19T20:25:59.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:58.989+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.991247+0000) 2024-02-19T20:25:59.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:59.046+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.048063+0000) 2024-02-19T20:25:59.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:59.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:59.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:59.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:59.693 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:59.503+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:25:59.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:59.309+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:59.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:59.309+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:59.694 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:59.309+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:25:59.694 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:59.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:59.694 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:59.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:59.694 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:59.479+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:25:59.695 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:25:59.479+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:59.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:25:59.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.725108+0000) 2024-02-19T20:25:59.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:25:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:25:59.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.883568+0000) 2024-02-19T20:26:00.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:00.048+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.050144+0000) 2024-02-19T20:26:00.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:00.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:26:00.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:00.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:26:00.431 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:00.270+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:26:00.696 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:00.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:26:00.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:00.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:26:00.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:00.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:26:00.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:00.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:00.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:00.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:00.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:00.523+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:00.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:00.523+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:00.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:00.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.883661+0000) 2024-02-19T20:26:00.973 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:00.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.725195+0000) 2024-02-19T20:26:01.416 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:01.051+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.053407+0000) 2024-02-19T20:26:01.416 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:01.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:26:01.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:01.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:26:01.417 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:01.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:28.691336+0000 (oldest deadline 2024-02-19T20:15:48.691336+0000) 2024-02-19T20:26:01.696 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:01.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:26:01.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:01.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:26:01.697 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:01.419+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:27.236755+0000 (oldest deadline 2024-02-19T20:15:47.236755+0000) 2024-02-19T20:26:01.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:01.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:01.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:01.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:01.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:01.522+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:01.697 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:01.522+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:01.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:01.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.883788+0000) 2024-02-19T20:26:01.973 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:01.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.725292+0000) 2024-02-19T20:26:02.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:02.054+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.055474+0000) 2024-02-19T20:26:02.878 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:02.723+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.725388+0000) 2024-02-19T20:26:02.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:02.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:02.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:02.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:02.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:02.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:02.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:02.491+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:03.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:02.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.883873+0000) 2024-02-19T20:26:03.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:03.054+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.056197+0000) 2024-02-19T20:26:03.839 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:03.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.725486+0000) 2024-02-19T20:26:03.839 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:03.533+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:03.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:03.533+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:03.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:03.533+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:03.840 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:03.533+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:04.115 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:04.000+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62246 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:04.115 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:03.994+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.995626+0000) 2024-02-19T20:26:04.116 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:03.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.883973+0000) 2024-02-19T20:26:04.116 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:03.994+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.995552+0000) 2024-02-19T20:26:04.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:03.997+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.998670+0000) 2024-02-19T20:26:04.117 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:04.054+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.056432+0000) 2024-02-19T20:26:04.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:04.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.725577+0000) 2024-02-19T20:26:04.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:04.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:04.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:04.581+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:04.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:04.582+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:04.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:04.582+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:05.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:04.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.884131+0000) 2024-02-19T20:26:05.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:05.057+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.058468+0000) 2024-02-19T20:26:05.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:05.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.725661+0000) 2024-02-19T20:26:05.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:05.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:05.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:05.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:05.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:05.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:05.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:05.537+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:06.200 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:05.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.884277+0000) 2024-02-19T20:26:06.200 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:06.059+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.060547+0000) 2024-02-19T20:26:06.806 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:06.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.725761+0000) 2024-02-19T20:26:06.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:06.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:06.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:06.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:06.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:06.500+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:06.807 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:06.501+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:07.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:06.882+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.884423+0000) 2024-02-19T20:26:07.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:07.061+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.062817+0000) 2024-02-19T20:26:07.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:07.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.725910+0000) 2024-02-19T20:26:07.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:07.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:07.879 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:07.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:07.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:07.532+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:07.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:07.532+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:08.202 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:07.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.884494+0000) 2024-02-19T20:26:08.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:08.062+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.063605+0000) 2024-02-19T20:26:08.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:08.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.726071+0000) 2024-02-19T20:26:08.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:08.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:08.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:08.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:08.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:08.580+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:08.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:08.580+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:09.202 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:09.001+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.002776+0000) 2024-02-19T20:26:09.202 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:09.006+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62321 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:09.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:09.003+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.005139+0000) 2024-02-19T20:26:09.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:09.062+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.063792+0000) 2024-02-19T20:26:09.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:08.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.884618+0000) 2024-02-19T20:26:09.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:09.001+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.002919+0000) 2024-02-19T20:26:09.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:09.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.726257+0000) 2024-02-19T20:26:09.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:09.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:09.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:09.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:09.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:09.594+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:09.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:09.595+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:10.174 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:10.064+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.066095+0000) 2024-02-19T20:26:10.174 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:09.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.884724+0000) 2024-02-19T20:26:10.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:10.724+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.726407+0000) 2024-02-19T20:26:10.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:10.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.884797+0000) 2024-02-19T20:26:10.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:10.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:10.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:10.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:10.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:10.634+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:10.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:10.634+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:11.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:11.067+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.068684+0000) 2024-02-19T20:26:11.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:11.725+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.726598+0000) 2024-02-19T20:26:11.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:11.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.884888+0000) 2024-02-19T20:26:11.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:11.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:11.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:11.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:11.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:11.654+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:11.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:11.654+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:12.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:12.069+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.071324+0000) 2024-02-19T20:26:12.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:12.725+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.726739+0000) 2024-02-19T20:26:12.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:12.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:12.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:12.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:12.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:12.624+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:12.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:12.624+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:13.202 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:13.070+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.072095+0000) 2024-02-19T20:26:13.202 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:12.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.884975+0000) 2024-02-19T20:26:13.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:13.725+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.726855+0000) 2024-02-19T20:26:13.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:13.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:13.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:13.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:13.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:13.593+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:13.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:13.593+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:14.137 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:14.013+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62393 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:14.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:14.008+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.010400+0000) 2024-02-19T20:26:14.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:13.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.885081+0000) 2024-02-19T20:26:14.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:14.008+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.009813+0000) 2024-02-19T20:26:14.138 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:14.010+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.011880+0000) 2024-02-19T20:26:14.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:14.070+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.072369+0000) 2024-02-19T20:26:14.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:14.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.885209+0000) 2024-02-19T20:26:14.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:14.725+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.727005+0000) 2024-02-19T20:26:14.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:14.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:14.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:14.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:14.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:14.638+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:14.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:14.638+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:15.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:15.072+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.074399+0000) 2024-02-19T20:26:15.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:15.725+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.727158+0000) 2024-02-19T20:26:15.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:15.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:15.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:15.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:15.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:15.596+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:15:39.891969+0000 (oldest deadline 2024-02-19T20:15:59.891969+0000) 2024-02-19T20:26:15.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:15.596+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:16.202 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:15.883+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.885300+0000) 2024-02-19T20:26:16.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:16.075+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.076492+0000) 2024-02-19T20:26:16.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:16.725+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.727332+0000) 2024-02-19T20:26:16.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:16.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.885474+0000) 2024-02-19T20:26:16.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:16.637+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:17.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:17.077+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.078626+0000) 2024-02-19T20:26:17.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:17.726+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.727461+0000) 2024-02-19T20:26:17.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:17.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.885598+0000) 2024-02-19T20:26:17.973 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:17.648+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:18.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:18.078+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.079484+0000) 2024-02-19T20:26:18.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:18.726+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.727571+0000) 2024-02-19T20:26:18.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:18.599+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:19.137 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:19.020+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62468 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:19.138 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:19.015+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.016572+0000) 2024-02-19T20:26:19.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:18.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.885702+0000) 2024-02-19T20:26:19.138 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:19.014+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.016402+0000) 2024-02-19T20:26:19.138 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:19.018+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.020138+0000) 2024-02-19T20:26:19.139 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:19.078+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.079732+0000) 2024-02-19T20:26:19.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:19.566+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:19.881 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:19.726+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.727667+0000) 2024-02-19T20:26:20.175 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:19.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.885794+0000) 2024-02-19T20:26:20.175 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:20.080+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.081868+0000) 2024-02-19T20:26:20.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:20.726+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.727767+0000) 2024-02-19T20:26:20.881 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:20.526+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:21.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:21.082+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.083958+0000) 2024-02-19T20:26:21.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:20.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.885867+0000) 2024-02-19T20:26:21.880 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:21.484+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:21.881 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:21.726+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.727876+0000) 2024-02-19T20:26:22.203 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:21.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.886007+0000) 2024-02-19T20:26:22.203 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:22.084+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.086061+0000) 2024-02-19T20:26:22.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:22.442+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:22.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:22.727+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.727987+0000) 2024-02-19T20:26:22.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:22.884+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.886143+0000) 2024-02-19T20:26:23.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:23.085+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.086938+0000) 2024-02-19T20:26:23.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:23.471+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:23.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:23.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:23.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:23.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:23.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:23.666+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:24.017 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:23.885+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.886248+0000) 2024-02-19T20:26:24.017 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:23.727+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.728079+0000) 2024-02-19T20:26:24.289 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:24.022+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.022900+0000) 2024-02-19T20:26:24.290 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:24.028+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62540 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:24.290 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:24.022+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.022759+0000) 2024-02-19T20:26:24.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:24.025+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.025943+0000) 2024-02-19T20:26:24.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:24.086+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.087128+0000) 2024-02-19T20:26:24.705 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:24.437+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:24.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:24.727+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.728234+0000) 2024-02-19T20:26:24.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:24.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:24.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:24.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:24.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:24.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:24.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:24.885+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.886405+0000) 2024-02-19T20:26:25.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:25.088+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.089298+0000) 2024-02-19T20:26:25.667 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:25.465+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:25.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:25.727+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.728363+0000) 2024-02-19T20:26:25.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:25.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:25.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:25.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:25.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:25.672+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:25.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:25.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.886503+0000) 2024-02-19T20:26:26.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:26.091+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.091482+0000) 2024-02-19T20:26:26.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:25.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:26.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:25.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:26.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:25.997+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:26.633 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:26.451+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:26.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:26.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.728519+0000) 2024-02-19T20:26:26.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:26.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:26.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:26.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:26.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:26.638+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:26.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:26.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.886611+0000) 2024-02-19T20:26:27.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:27.093+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.093531+0000) 2024-02-19T20:26:27.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:27.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:27.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.012+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:27.657 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:27.488+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:27.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.728607+0000) 2024-02-19T20:26:27.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:27.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:27.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:27.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:27.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:27.662+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:27.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:27.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.886751+0000) 2024-02-19T20:26:28.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:28.093+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.094277+0000) 2024-02-19T20:26:28.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:28.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:28.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:27.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:28.662 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:28.475+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:28.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:28.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.728703+0000) 2024-02-19T20:26:28.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:28.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:28.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:28.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:28.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:28.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:28.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:28.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.886850+0000) 2024-02-19T20:26:29.290 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:29.029+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.029796+0000) 2024-02-19T20:26:29.290 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:29.034+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62614 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:29.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:29.032+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.032518+0000) 2024-02-19T20:26:29.291 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:29.094+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.094529+0000) 2024-02-19T20:26:29.291 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:29.291 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:29.291 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.006+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:29.291 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.029+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.029636+0000) 2024-02-19T20:26:29.637 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:29.507+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:29.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.728799+0000) 2024-02-19T20:26:29.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:29.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:29.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:29.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:29.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:29.642+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:29.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:29.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.886957+0000) 2024-02-19T20:26:30.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:30.096+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.096491+0000) 2024-02-19T20:26:30.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:30.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:30.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:29.979+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:30.631 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:30.469+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:30.881 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:30.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.728883+0000) 2024-02-19T20:26:30.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:30.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:30.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:30.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:30.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:30.636+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:31.150 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:30.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.887053+0000) 2024-02-19T20:26:31.150 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:31.098+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.098491+0000) 2024-02-19T20:26:31.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:30.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:31.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:30.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:31.151 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:30.990+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:31.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:31.434+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:31.671+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:31.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:31.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.887165+0000) 2024-02-19T20:26:31.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:31.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.728984+0000) 2024-02-19T20:26:32.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:32.100+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.100719+0000) 2024-02-19T20:26:32.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:32.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:32.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.010+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:32.686 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:32.455+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:32.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.729078+0000) 2024-02-19T20:26:32.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:32.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:32.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:32.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:32.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:32.691+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:32.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:32.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.887276+0000) 2024-02-19T20:26:33.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:33.101+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.101442+0000) 2024-02-19T20:26:33.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:33.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:33.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:32.982+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:33.702 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:33.489+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:33.954 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:33.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.729188+0000) 2024-02-19T20:26:33.955 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:33.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:33.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:33.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:33.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:33.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:33.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:33.708+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:33.955 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:33.886+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.887380+0000) 2024-02-19T20:26:34.222 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:34.036+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.036521+0000) 2024-02-19T20:26:34.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:34.038+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.038707+0000) 2024-02-19T20:26:34.222 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:34.101+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.101661+0000) 2024-02-19T20:26:34.223 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:34.041+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62687 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:34.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:33.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:34.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:33.959+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:34.223 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:34.035+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.036168+0000) 2024-02-19T20:26:34.881 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:34.728+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.729349+0000) 2024-02-19T20:26:34.882 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:34.531+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:34.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:34.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:34.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:34.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:34.882 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:34.755+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:35.205 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:34.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.887462+0000) 2024-02-19T20:26:35.206 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:35.103+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.103689+0000) 2024-02-19T20:26:35.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:35.003+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:35.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:35.003+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:35.206 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:35.003+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:35.780 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:35.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.729503+0000) 2024-02-19T20:26:35.781 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:35.523+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:36.036 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:35.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:36.037 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:35.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:36.037 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:35.785+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:36.037 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:35.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.887558+0000) 2024-02-19T20:26:36.294 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:36.105+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.106334+0000) 2024-02-19T20:26:36.295 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:36.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:36.295 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:36.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:36.295 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:36.040+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:36.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:36.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:36.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:36.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:36.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:36.525+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:36.723 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:36.525+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:36.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:36.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.729634+0000) 2024-02-19T20:26:36.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:36.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:36.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:36.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:36.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:36.773+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:36.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:36.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.887677+0000) 2024-02-19T20:26:37.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:37.108+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.108846+0000) 2024-02-19T20:26:37.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:37.047+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:37.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:37.047+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:37.473 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:37.047+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:37.816 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:37.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.729727+0000) 2024-02-19T20:26:37.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:37.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:37.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:37.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:37.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:37.538+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:37.817 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:37.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:38.104 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:38.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:38.104 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:38.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:38.104 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:38.067+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:38.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:37.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:38.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:37.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:38.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:37.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:38.105 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:37.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.887822+0000) 2024-02-19T20:26:38.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:38.109+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.109794+0000) 2024-02-19T20:26:38.800 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:38.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.729828+0000) 2024-02-19T20:26:38.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:38.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:38.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:38.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:38.800 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:38.521+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:38.801 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:38.521+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:39.050 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:39.047+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62761 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:39.051 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:39.045+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.046364+0000) 2024-02-19T20:26:39.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:39.042+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.042964+0000) 2024-02-19T20:26:39.051 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:38.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:39.051 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:38.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:39.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:38.805+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:39.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:38.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.887926+0000) 2024-02-19T20:26:39.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:39.043+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.043503+0000) 2024-02-19T20:26:39.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:39.109+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.110030+0000) 2024-02-19T20:26:39.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:39.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:39.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:39.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:39.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:39.069+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:39.822 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:39.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.729931+0000) 2024-02-19T20:26:39.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:39.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:39.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:39.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:39.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:39.537+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:39.823 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:39.538+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:40.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:39.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:40.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:39.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:40.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:39.828+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:40.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:39.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.888067+0000) 2024-02-19T20:26:40.365 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:40.113+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.113483+0000) 2024-02-19T20:26:40.366 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:40.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:40.366 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:40.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:40.366 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:40.092+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:40.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:40.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:40.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:40.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:40.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:40.519+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:40.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:40.519+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:41.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:40.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.730014+0000) 2024-02-19T20:26:41.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:41.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:41.041 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:41.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:41.041 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:40.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:41.041 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:40.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:41.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:40.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:41.042 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:40.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.888231+0000) 2024-02-19T20:26:41.290 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:41.046+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:41.291 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:41.116+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.116962+0000) 2024-02-19T20:26:41.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:41.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:41.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:41.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:41.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:41.498+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:41.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:41.498+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:42.014 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:41.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.730155+0000) 2024-02-19T20:26:42.015 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:42.015 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:42.015 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:41.861+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:42.015 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:41.861+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:42.015 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:41.861+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:42.015 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:41.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.888377+0000) 2024-02-19T20:26:42.291 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.019+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:42.291 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:42.120+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.120523+0000) 2024-02-19T20:26:42.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:42.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:42.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:42.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:42.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:42.491+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:42.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:42.491+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:42.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:42.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:42.881+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:42.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.888466+0000) 2024-02-19T20:26:43.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.730276+0000) 2024-02-19T20:26:43.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:43.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:43.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:42.971+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:43.456 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:43.121+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.121644+0000) 2024-02-19T20:26:43.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:43.462+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:43.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:43.462+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:43.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:43.462+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:43.457 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:43.462+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:44.044 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:43.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.730401+0000) 2024-02-19T20:26:44.044 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:43.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:44.044 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:43.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:44.044 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:43.936+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:44.045 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:43.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:44.045 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:43.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:44.045 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:43.882+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:44.045 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:43.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.888575+0000) 2024-02-19T20:26:44.045 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:44.049+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.049615+0000) 2024-02-19T20:26:44.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:44.049+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.049773+0000) 2024-02-19T20:26:44.469 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:44.055+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62834 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:44.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:44.052+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.052864+0000) 2024-02-19T20:26:44.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:44.121+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.121887+0000) 2024-02-19T20:26:44.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:44.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:44.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:44.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:44.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:44.474+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:45.047 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:44.474+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:45.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:44.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.888710+0000) 2024-02-19T20:26:45.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:44.923+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:45.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:44.923+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:45.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:44.923+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:45.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:44.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.730464+0000) 2024-02-19T20:26:45.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:44.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:45.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:44.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:45.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:44.928+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:45.441 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:45.123+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.124076+0000) 2024-02-19T20:26:45.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:45.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:45.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:45.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:45.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:45.446+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:45.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:45.446+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:46.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:45.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.730497+0000) 2024-02-19T20:26:46.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:45.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:46.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:45.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:46.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:45.926+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:46.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:45.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.888800+0000) 2024-02-19T20:26:46.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:45.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:46.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:45.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:46.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:45.935+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:46.399 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:46.127+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.127712+0000) 2024-02-19T20:26:46.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:46.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:46.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:46.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:46.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:46.404+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:46.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:46.404+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:47.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:46.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.730598+0000) 2024-02-19T20:26:47.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:46.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:47.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:46.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:47.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:46.892+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:47.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:46.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:47.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:46.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:47.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:46.886+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:47.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:46.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.888889+0000) 2024-02-19T20:26:47.412 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:47.128+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.129752+0000) 2024-02-19T20:26:47.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:47.416+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:47.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:47.416+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:47.413 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:47.416+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:47.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:47.417+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:48.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:47.885+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:48.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:47.885+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:48.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:47.885+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:48.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:47.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.889002+0000) 2024-02-19T20:26:48.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:47.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.730690+0000) 2024-02-19T20:26:48.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:47.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:48.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:47.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:48.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:47.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:48.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:48.129+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.130916+0000) 2024-02-19T20:26:48.385 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:48.390+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:48.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:48.390+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:48.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:48.390+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:48.386 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:48.390+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:49.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:48.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.889096+0000) 2024-02-19T20:26:49.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:48.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:49.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:48.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:49.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:48.899+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:49.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:48.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.730802+0000) 2024-02-19T20:26:49.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:48.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:49.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:48.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:49.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:48.842+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:49.391 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:49.061+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62911 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:49.392 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:49.055+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.057132+0000) 2024-02-19T20:26:49.392 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:49.055+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.056577+0000) 2024-02-19T20:26:49.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:49.058+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.060221+0000) 2024-02-19T20:26:49.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:49.129+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.131108+0000) 2024-02-19T20:26:49.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:49.395+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:49.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:49.395+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:49.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:49.395+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:49.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:49.396+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:50.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:49.866+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:50.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:49.866+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:50.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:49.866+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:50.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:49.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.889203+0000) 2024-02-19T20:26:50.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:49.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.730955+0000) 2024-02-19T20:26:50.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:49.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:50.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:49.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:50.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:49.887+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:50.362 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:50.131+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.133115+0000) 2024-02-19T20:26:50.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:50.367+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:50.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:50.367+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:50.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:50.367+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:50.363 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:50.367+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:51.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:50.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:51.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:50.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:51.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:50.835+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:51.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:50.887+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.889351+0000) 2024-02-19T20:26:51.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:50.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.731096+0000) 2024-02-19T20:26:51.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:50.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:51.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:50.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:51.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:50.904+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:51.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:51.133+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.135269+0000) 2024-02-19T20:26:51.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:51.327+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:51.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:51.327+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:51.323 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:51.327+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:51.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:51.327+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:52.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:51.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:52.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:51.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:52.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:51.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:52.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:51.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.889465+0000) 2024-02-19T20:26:52.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:51.729+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.731270+0000) 2024-02-19T20:26:52.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:51.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:52.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:51.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:52.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:51.866+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:52.337 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:52.136+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.137540+0000) 2024-02-19T20:26:52.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:52.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:52.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:52.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:52.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:52.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:52.653 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:52.342+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:52.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:52.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.731431+0000) 2024-02-19T20:26:52.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:52.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:52.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:52.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:52.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:52.844+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:52.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:52.874+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:52.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:52.874+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:52.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:52.874+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:52.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:52.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.889558+0000) 2024-02-19T20:26:53.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:53.137+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.138712+0000) 2024-02-19T20:26:53.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:53.370+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:53.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:53.370+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:53.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:53.370+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:53.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:53.370+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:54.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:53.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.889656+0000) 2024-02-19T20:26:54.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:53.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:54.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:53.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:54.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:53.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:54.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:53.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.731594+0000) 2024-02-19T20:26:54.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:53.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:54.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:53.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:54.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:53.826+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:54.386 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:54.068+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 62984 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:54.387 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:54.062+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.064055+0000) 2024-02-19T20:26:54.387 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:54.062+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.063624+0000) 2024-02-19T20:26:54.387 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:54.065+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.066510+0000) 2024-02-19T20:26:54.387 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:54.137+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.138933+0000) 2024-02-19T20:26:54.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:54.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:54.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:54.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:54.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:54.391+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:54.388 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:54.391+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:55.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:54.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:55.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:54.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:55.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:54.871+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:55.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:54.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.889750+0000) 2024-02-19T20:26:55.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:54.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.731750+0000) 2024-02-19T20:26:55.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:54.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:55.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:54.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:55.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:54.822+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:55.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:55.139+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.140969+0000) 2024-02-19T20:26:55.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:55.382+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:55.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:55.382+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:55.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:55.382+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:55.378 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:55.382+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:56.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:55.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.731827+0000) 2024-02-19T20:26:56.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:55.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:56.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:55.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:56.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:55.808+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:56.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:55.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.889828+0000) 2024-02-19T20:26:56.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:55.904+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:56.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:55.904+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:56.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:55.904+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:56.406 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:56.141+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.143137+0000) 2024-02-19T20:26:56.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:56.410+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:56.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:56.410+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:56.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:56.410+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:56.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:56.410+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:57.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:56.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.731926+0000) 2024-02-19T20:26:57.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:56.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:57.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:56.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:57.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:56.827+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:57.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:56.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.889943+0000) 2024-02-19T20:26:57.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:56.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:57.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:56.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:57.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:56.926+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:57.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:57.144+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.145578+0000) 2024-02-19T20:26:57.439 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:57.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:57.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:57.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:57.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:57.443+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:57.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:57.443+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:58.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:57.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.890054+0000) 2024-02-19T20:26:58.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:57.934+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:58.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:57.934+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:58.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:57.934+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:58.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:57.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.732041+0000) 2024-02-19T20:26:58.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:57.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:58.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:57.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:58.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:57.861+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:58.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:58.145+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.146684+0000) 2024-02-19T20:26:58.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:58.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:58.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:58.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:58.448 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:58.452+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:58.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:58.453+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:59.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:58.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.890144+0000) 2024-02-19T20:26:59.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:58.903+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:59.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:58.903+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:59.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:58.903+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:26:59.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:58.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.732146+0000) 2024-02-19T20:26:59.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:58.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:59.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:58.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:59.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:58.847+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:26:59.402 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:26:59.075+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63058 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:26:59.402 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:59.070+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.071978+0000) 2024-02-19T20:26:59.402 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:59.069+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.071227+0000) 2024-02-19T20:26:59.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:59.072+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.074083+0000) 2024-02-19T20:26:59.403 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:59.145+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.147015+0000) 2024-02-19T20:26:59.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:59.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:59.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:59.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:59.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:59.406+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:26:59.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:26:59.406+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:00.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:59.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.732260+0000) 2024-02-19T20:27:00.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:59.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:00.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:59.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:00.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:26:59.857+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:00.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:59.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:00.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:59.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:00.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:59.862+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:00.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:26:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:26:59.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.890258+0000) 2024-02-19T20:27:00.372 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:00.147+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.149215+0000) 2024-02-19T20:27:00.373 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:00.377+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:00.373 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:00.377+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:00.373 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:00.377+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:00.373 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:00.377+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:01.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:00.875+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:01.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:00.875+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:01.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:00.875+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:01.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:00.888+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.890357+0000) 2024-02-19T20:27:01.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:00.730+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.732337+0000) 2024-02-19T20:27:01.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:00.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:01.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:00.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:01.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:00.897+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:01.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:01.149+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.151313+0000) 2024-02-19T20:27:01.342 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:01.346+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:01.342 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:01.346+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:01.631 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:01.346+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:01.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:01.346+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:01.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:01.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.732450+0000) 2024-02-19T20:27:01.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:01.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:01.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:01.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:01.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:01.884+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:02.147 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:01.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.890482+0000) 2024-02-19T20:27:02.147 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:01.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:02.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:01.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:02.148 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:01.920+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:02.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:02.152+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.153568+0000) 2024-02-19T20:27:02.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:02.360+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:02.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:02.360+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:02.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:02.360+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:02.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:02.360+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:03.048 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:02.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.732539+0000) 2024-02-19T20:27:03.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:02.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:03.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:02.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:03.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:02.858+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:03.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:02.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:03.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:02.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:03.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:02.883+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:03.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:02.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.890571+0000) 2024-02-19T20:27:03.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:03.153+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.154466+0000) 2024-02-19T20:27:03.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:03.311+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:03.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:03.311+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:03.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:03.311+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:03.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:03.311+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:04.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:03.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.890675+0000) 2024-02-19T20:27:04.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:03.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:04.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:03.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:04.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:03.933+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:04.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:03.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.732646+0000) 2024-02-19T20:27:04.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:03.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:04.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:03.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:04.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:03.824+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:04.471 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:04.082+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63131 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:04.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:04.076+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.078356+0000) 2024-02-19T20:27:04.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:04.076+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.077882+0000) 2024-02-19T20:27:04.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:04.080+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.081860+0000) 2024-02-19T20:27:04.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:04.153+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.154620+0000) 2024-02-19T20:27:04.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:04.271+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:04.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:04.271+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:04.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:04.271+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:04.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:04.271+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:05.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:04.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.890828+0000) 2024-02-19T20:27:05.049 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:04.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:05.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:04.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:05.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:04.897+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:05.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:04.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.732817+0000) 2024-02-19T20:27:05.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:04.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:05.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:04.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:05.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:04.787+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:05.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:05.156+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.158010+0000) 2024-02-19T20:27:05.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:05.315+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:05.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:05.315+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:05.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:05.315+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:05.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:05.315+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:06.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:05.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.732920+0000) 2024-02-19T20:27:06.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:05.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:06.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:05.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:06.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:05.777+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:06.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:05.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:06.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:05.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:06.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:05.854+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:06.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:05.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.890915+0000) 2024-02-19T20:27:06.322 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:06.158+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.160093+0000) 2024-02-19T20:27:06.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:06.326+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:06.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:06.326+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:06.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:06.326+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:06.669 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:06.326+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:06.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:06.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:06.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:06.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:06.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:06.730+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:06.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:06.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.733111+0000) 2024-02-19T20:27:06.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:06.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:06.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:06.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:06.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:06.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:06.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:06.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.891047+0000) 2024-02-19T20:27:07.337 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:07.162+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.163533+0000) 2024-02-19T20:27:07.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:07.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:07.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:07.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:07.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:07.342+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:07.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:07.342+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:08.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:07.731+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.733288+0000) 2024-02-19T20:27:08.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:07.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:08.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:07.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:08.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:07.764+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:08.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:07.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:08.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:07.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:08.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:07.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:08.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:07.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.891233+0000) 2024-02-19T20:27:08.345 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:08.162+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.164298+0000) 2024-02-19T20:27:08.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:08.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:08.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:08.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:08.721 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:08.350+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:08.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:08.350+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:09.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:08.732+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.733483+0000) 2024-02-19T20:27:09.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:08.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:09.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:08.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:09.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:08.769+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:09.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:08.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:09.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:08.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:09.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:08.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:09.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:08.889+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.891381+0000) 2024-02-19T20:27:09.364 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:09.083+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.085290+0000) 2024-02-19T20:27:09.365 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:09.088+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63205 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:09.365 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:09.083+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.085110+0000) 2024-02-19T20:27:09.365 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:09.087+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.089238+0000) 2024-02-19T20:27:09.365 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:09.163+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.164492+0000) 2024-02-19T20:27:09.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:09.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:09.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:09.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:09.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:09.369+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:09.366 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:09.369+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:10.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:09.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:10.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:09.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:10.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:09.872+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:10.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:09.890+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.891497+0000) 2024-02-19T20:27:10.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:09.732+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.733609+0000) 2024-02-19T20:27:10.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:09.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:10.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:09.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:10.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:09.792+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:10.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:10.165+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.166910+0000) 2024-02-19T20:27:10.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:10.324+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:10.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:10.324+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:10.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:10.324+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:10.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:10.324+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:11.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:10.732+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.733687+0000) 2024-02-19T20:27:11.049 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:10.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:11.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:10.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:11.050 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:10.743+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:11.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:10.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:11.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:10.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:11.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:10.843+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:11.050 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:10.890+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.891593+0000) 2024-02-19T20:27:11.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:11.169+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.170441+0000) 2024-02-19T20:27:11.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:11.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:11.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:11.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:11.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:11.286+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:11.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:11.286+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:11.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:11.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:11.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:11.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:11.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:11.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:11.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:11.732+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.733776+0000) 2024-02-19T20:27:11.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:11.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:11.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:11.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:11.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:11.844+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:11.973 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:11.890+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.891704+0000) 2024-02-19T20:27:12.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:12.171+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.173136+0000) 2024-02-19T20:27:12.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:12.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:12.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:12.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:12.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:12.268+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:12.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:12.268+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:12.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:12.702+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:12.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:12.702+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:12.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:12.702+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:12.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:12.732+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.733869+0000) 2024-02-19T20:27:12.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:12.841+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:12.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:12.841+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:12.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:12.841+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:12.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:12.890+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.891839+0000) 2024-02-19T20:27:13.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:13.172+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.174010+0000) 2024-02-19T20:27:13.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:13.249+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:13.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:13.249+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:13.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:13.249+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:13.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:13.249+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:13.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:13.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:13.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:13.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:13.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:13.696+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:13.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:13.733+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.733981+0000) 2024-02-19T20:27:13.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:13.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:13.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:13.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:13.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:13.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:13.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:13.890+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.891987+0000) 2024-02-19T20:27:14.471 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:14.095+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63278 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:14.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:14.089+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.091430+0000) 2024-02-19T20:27:14.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:14.090+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.092386+0000) 2024-02-19T20:27:14.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:14.092+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.094279+0000) 2024-02-19T20:27:14.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:14.173+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.174217+0000) 2024-02-19T20:27:14.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:14.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:14.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:14.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:14.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:14.217+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:14.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:14.217+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:14.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:14.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:14.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:14.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:14.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:14.684+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:14.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:14.732+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.734068+0000) 2024-02-19T20:27:14.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:14.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:14.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:14.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:14.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:14.772+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:14.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:14.891+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.892121+0000) 2024-02-19T20:27:15.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:15.176+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.177817+0000) 2024-02-19T20:27:15.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:15.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:15.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:15.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:15.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:15.183+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:15.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:15.183+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:15.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:15.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:15.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:15.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:15.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:15.690+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:15.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:15.733+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.734164+0000) 2024-02-19T20:27:15.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:15.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:15.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:15.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:15.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:15.730+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:15.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:15.891+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.892214+0000) 2024-02-19T20:27:16.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:16.179+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.179837+0000) 2024-02-19T20:27:16.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:16.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:16.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:16.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:16.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:16.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:16.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:16.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:16.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:16.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:16.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:16.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:16.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:16.720+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:16.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:16.733+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.734273+0000) 2024-02-19T20:27:16.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:16.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:16.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:16.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:16.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:16.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:16.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:16.891+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.892320+0000) 2024-02-19T20:27:17.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:17.182+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.183349+0000) 2024-02-19T20:27:17.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:17.262+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:17.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:17.262+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:17.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:17.262+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:17.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:17.262+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:17.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:17.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:17.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:17.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:17.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:17.673+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:17.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:17.733+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.734369+0000) 2024-02-19T20:27:17.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:17.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:17.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:17.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:17.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:17.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:17.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:17.891+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.892431+0000) 2024-02-19T20:27:18.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:18.184+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.184635+0000) 2024-02-19T20:27:18.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:18.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:18.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:18.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:18.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:18.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:18.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:18.256+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:18.886 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:18.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:18.886 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:18.634+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:18.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:18.635+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:18.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:18.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.734492+0000) 2024-02-19T20:27:18.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:18.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:18.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:18.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:18.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:18.699+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:19.178 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:19.103+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63345 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:19.179 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:19.099+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.100107+0000) 2024-02-19T20:27:19.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:19.098+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.098788+0000) 2024-02-19T20:27:19.179 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:18.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.892523+0000) 2024-02-19T20:27:19.179 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:19.098+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.098591+0000) 2024-02-19T20:27:19.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:19.184+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.184806+0000) 2024-02-19T20:27:19.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:19.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:19.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:19.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:19.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:19.264+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:19.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:19.264+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:19.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:19.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:19.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:19.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:19.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:19.667+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:19.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:19.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.734604+0000) 2024-02-19T20:27:19.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:19.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:19.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:19.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:19.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:19.667+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:19.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:19.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.892611+0000) 2024-02-19T20:27:20.301 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:20.186+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.187173+0000) 2024-02-19T20:27:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:20.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:20.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:20.258+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:20.259+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:20.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:20.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:20.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:20.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:20.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:20.709+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:20.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:20.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:20.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:20.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:20.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:20.627+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:20.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:20.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.734703+0000) 2024-02-19T20:27:21.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:20.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.892703+0000) 2024-02-19T20:27:21.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:21.188+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.189268+0000) 2024-02-19T20:27:21.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:21.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:21.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:21.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:21.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:21.220+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:21.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:21.220+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:21.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:21.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:21.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:21.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:21.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:21.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:21.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:21.636+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:21.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:21.636+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:21.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:21.636+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:21.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:21.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.734837+0000) 2024-02-19T20:27:22.184 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:21.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.892818+0000) 2024-02-19T20:27:22.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:22.191+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.191456+0000) 2024-02-19T20:27:22.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:22.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:22.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:22.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:22.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:22.219+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:22.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:22.219+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:22.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:22.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:22.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:22.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:22.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:22.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:22.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:22.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.892912+0000) 2024-02-19T20:27:22.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:22.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:22.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:22.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:22.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:22.659+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:22.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:22.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.734984+0000) 2024-02-19T20:27:23.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:23.191+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.192299+0000) 2024-02-19T20:27:23.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:23.244+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:23.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:23.244+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:23.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:23.244+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:23.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:23.244+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:23.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:23.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:23.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:23.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:23.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:23.775+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:23.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:23.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.893046+0000) 2024-02-19T20:27:23.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:23.649+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:23.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:23.649+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:23.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:23.649+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:23.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:23.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.735136+0000) 2024-02-19T20:27:24.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:24.105+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.105886+0000) 2024-02-19T20:27:24.471 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:24.110+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63403 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:24.472 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:24.104+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.105382+0000) 2024-02-19T20:27:24.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:24.107+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.107665+0000) 2024-02-19T20:27:24.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:24.192+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.192533+0000) 2024-02-19T20:27:24.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:24.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:24.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:24.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:24.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:24.256+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:24.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:24.256+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:24.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:24.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:24.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:24.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:24.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:24.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:24.972 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:24.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.893188+0000) 2024-02-19T20:27:24.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:24.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:24.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:24.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:24.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:24.680+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:24.973 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:24.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.735269+0000) 2024-02-19T20:27:25.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:25.194+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.195095+0000) 2024-02-19T20:27:25.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:25.296+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:25.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:25.296+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:25.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:25.296+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:25.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:25.296+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:25.886 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:25.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:25.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:25.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:25.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:25.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:25.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:25.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:25.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:25.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:25.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:25.641+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:25.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:25.734+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.735366+0000) 2024-02-19T20:27:26.190 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:25.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.893267+0000) 2024-02-19T20:27:26.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:26.196+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.197116+0000) 2024-02-19T20:27:26.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:26.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:26.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:26.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:26.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:26.294+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:26.445 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:26.294+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:26.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:26.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:26.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:26.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:26.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:26.616+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:26.971 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:26.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.735478+0000) 2024-02-19T20:27:26.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:26.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:26.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:26.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:26.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:26.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:26.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:26.892+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.893368+0000) 2024-02-19T20:27:27.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:27.199+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.199707+0000) 2024-02-19T20:27:27.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:27.255+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:27.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:27.255+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:27.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:27.255+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:27.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:27.255+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:27.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:27.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:27.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:27.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:27.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:27.760+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:27.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:27.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.893461+0000) 2024-02-19T20:27:27.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:27.644+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:27.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:27.644+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:27.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:27.644+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:27.972 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:27.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.735601+0000) 2024-02-19T20:27:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:28.200+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.200683+0000) 2024-02-19T20:27:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:28.304+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:28.304+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:28.304+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:28.304+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:28.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:28.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:28.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:28.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:28.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:28.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:28.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:28.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:28.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:28.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:28.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:28.609+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:28.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:28.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.735712+0000) 2024-02-19T20:27:29.194 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:29.116+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63462 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:29.194 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:29.114+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.115381+0000) 2024-02-19T20:27:29.195 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:29.111+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.111806+0000) 2024-02-19T20:27:29.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:28.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.893550+0000) 2024-02-19T20:27:29.195 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:29.112+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.112984+0000) 2024-02-19T20:27:29.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:29.200+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.200861+0000) 2024-02-19T20:27:29.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:29.330+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:29.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:29.330+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:29.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:29.330+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:29.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:29.330+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:29.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:29.561+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:29.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:29.561+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:29.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:29.561+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:29.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:29.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.735808+0000) 2024-02-19T20:27:29.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:29.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:29.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:29.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:29.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:29.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:30.178 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:29.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.893662+0000) 2024-02-19T20:27:30.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:30.203+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.203597+0000) 2024-02-19T20:27:30.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:30.305+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:30.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:30.305+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:30.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:30.305+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:30.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:30.305+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:30.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:30.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:30.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:30.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:30.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:30.696+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:30.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:30.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:30.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:30.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:30.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:30.591+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:30.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:30.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.735882+0000) 2024-02-19T20:27:31.198 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:30.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.893742+0000) 2024-02-19T20:27:31.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:31.205+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.205591+0000) 2024-02-19T20:27:31.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:31.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:31.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:31.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:31.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:31.261+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:31.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:31.262+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:31.867 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:31.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:31.867 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:31.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:31.867 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:31.700+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:31.867 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:31.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:31.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:31.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:31.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:31.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:31.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:31.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.736034+0000) 2024-02-19T20:27:32.201 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:31.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.893840+0000) 2024-02-19T20:27:32.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:32.207+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.207851+0000) 2024-02-19T20:27:32.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:32.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:32.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:32.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:32.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:32.213+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:32.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:32.213+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:32.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:32.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:32.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:32.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:32.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:32.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:32.887 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:32.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:32.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:32.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:32.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:32.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:32.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:32.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.736206+0000) 2024-02-19T20:27:33.202 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:32.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.893933+0000) 2024-02-19T20:27:33.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:33.208+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.209066+0000) 2024-02-19T20:27:33.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:33.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:33.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:33.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:33.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:33.218+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:33.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:33.218+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:33.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:33.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:33.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:33.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:33.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:33.536+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:33.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:33.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.736362+0000) 2024-02-19T20:27:33.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:33.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:33.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:33.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:33.844 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:33.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:34.111 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:33.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.894079+0000) 2024-02-19T20:27:34.470 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:34.117+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.118403+0000) 2024-02-19T20:27:34.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:34.118+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.119118+0000) 2024-02-19T20:27:34.471 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:34.123+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63520 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:34.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:34.122+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.122907+0000) 2024-02-19T20:27:34.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:34.208+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.209279+0000) 2024-02-19T20:27:34.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:34.266+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:34.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:34.266+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:34.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:34.266+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:34.473 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:34.266+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:34.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:34.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:34.887 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:34.680+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:34.888 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:34.681+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:34.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:34.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:34.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:34.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:34.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:34.545+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:34.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:34.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.736474+0000) 2024-02-19T20:27:35.204 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:34.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.894213+0000) 2024-02-19T20:27:35.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:35.210+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.211412+0000) 2024-02-19T20:27:35.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:35.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:35.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:35.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:35.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:35.274+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:35.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:35.274+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:35.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:35.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:35.772 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:35.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:35.773 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:35.665+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:35.773 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:35.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:35.773 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:35.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:35.773 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:35.503+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:35.773 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:35.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.736553+0000) 2024-02-19T20:27:36.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:35.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.894309+0000) 2024-02-19T20:27:36.463 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:36.213+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.213479+0000) 2024-02-19T20:27:36.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:36.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:36.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:36.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:36.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:36.240+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:36.464 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:36.240+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:36.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:36.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:36.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:36.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:36.722 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:36.686+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:36.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:36.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:36.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:36.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:36.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:36.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:37.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:36.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.894417+0000) 2024-02-19T20:27:37.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:36.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.736645+0000) 2024-02-19T20:27:37.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:37.192+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:37.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:37.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:37.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:37.193+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:37.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:37.193+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:37.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:37.215+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.215639+0000) 2024-02-19T20:27:37.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:37.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:37.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:37.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:37.730 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:37.483+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:37.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:37.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:37.730 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:37.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:37.731 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:37.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:38.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:37.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.894480+0000) 2024-02-19T20:27:38.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:37.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.736742+0000) 2024-02-19T20:27:38.468 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:38.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:38.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:38.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:38.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:38.164+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:38.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:38.164+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:38.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:38.215+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.216583+0000) 2024-02-19T20:27:38.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:38.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:38.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:38.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:38.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:38.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:38.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:38.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.736924+0000) 2024-02-19T20:27:38.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:38.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:38.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:38.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:38.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:38.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:38.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:38.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.894600+0000) 2024-02-19T20:27:39.420 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:39.129+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63579 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:39.421 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:39.125+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.126622+0000) 2024-02-19T20:27:39.421 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:39.124+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.126195+0000) 2024-02-19T20:27:39.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:39.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:39.422 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:39.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:39.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:39.126+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.128342+0000) 2024-02-19T20:27:39.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:39.133+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:39.422 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:39.133+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:39.133+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:39.133+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:39.423 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:39.215+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.216788+0000) 2024-02-19T20:27:39.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:39.426+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:40.053 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:39.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.737100+0000) 2024-02-19T20:27:40.053 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:39.748+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:40.053 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:39.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:40.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:39.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:40.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:39.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.894687+0000) 2024-02-19T20:27:40.429 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:40.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:40.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:40.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:40.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:40.084+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:40.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:40.084+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:40.430 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:40.217+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.218824+0000) 2024-02-19T20:27:40.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:40.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:40.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:40.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:40.698 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:40.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:40.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:40.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.737239+0000) 2024-02-19T20:27:40.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:40.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:40.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:40.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:40.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:40.703+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:40.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:40.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.894782+0000) 2024-02-19T20:27:41.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:41.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:41.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:41.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:41.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:41.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:41.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:41.048+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:41.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:41.219+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.221008+0000) 2024-02-19T20:27:41.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:41.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:41.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:41.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:41.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:41.458+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:42.001 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:41.735+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.737413+0000) 2024-02-19T20:27:42.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:41.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:42.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:41.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:42.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:41.739+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:42.001 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:41.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.894891+0000) 2024-02-19T20:27:42.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:42.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:42.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:42.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:42.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:42.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:42.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:42.006+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:42.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:42.222+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.223535+0000) 2024-02-19T20:27:42.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:42.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:42.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:42.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:42.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:42.422+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:43.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:42.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.737554+0000) 2024-02-19T20:27:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:42.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:42.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:43.047 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:42.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:43.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:42.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.895025+0000) 2024-02-19T20:27:43.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:43.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:43.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:43.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:43.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:43.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:43.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:43.052+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:43.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:43.223+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.224743+0000) 2024-02-19T20:27:43.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:43.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:43.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:43.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:43.699 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:43.470+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:43.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:43.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.737667+0000) 2024-02-19T20:27:43.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:43.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:43.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:43.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:43.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:43.704+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:43.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:43.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.895185+0000) 2024-02-19T20:27:44.297 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:44.136+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63637 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:44.297 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:44.131+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.132834+0000) 2024-02-19T20:27:44.298 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:44.130+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.132407+0000) 2024-02-19T20:27:44.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:44.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:44.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:44.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:44.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:44.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:44.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:44.024+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:44.299 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:44.133+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.134715+0000) 2024-02-19T20:27:44.299 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:44.223+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.224898+0000) 2024-02-19T20:27:44.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:44.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:44.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:44.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:44.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:44.456+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:45.018 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:44.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.737794+0000) 2024-02-19T20:27:45.018 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:44.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:45.018 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:44.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:45.018 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:44.740+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:45.018 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:44.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.895292+0000) 2024-02-19T20:27:45.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:45.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:45.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.023+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:45.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.023+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:45.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.226+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.228366+0000) 2024-02-19T20:27:45.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:45.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:45.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:45.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:45.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:45.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:45.993 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:45.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.737871+0000) 2024-02-19T20:27:45.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:45.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:45.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:45.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:45.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:45.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:45.994 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:45.893+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.895385+0000) 2024-02-19T20:27:46.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:46.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:46.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:46.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:45.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:46.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:46.229+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.230487+0000) 2024-02-19T20:27:46.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:46.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:46.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:46.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:46.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:46.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:46.983 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:46.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.738004+0000) 2024-02-19T20:27:46.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:46.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:46.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:46.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:46.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:46.810+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:46.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:46.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.895482+0000) 2024-02-19T20:27:47.297 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:46.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:47.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:46.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:47.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:46.993+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:47.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:46.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:47.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:47.231+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.232592+0000) 2024-02-19T20:27:47.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:47.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:47.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:47.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:47.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:47.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:48.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:47.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.738147+0000) 2024-02-19T20:27:48.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:47.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:48.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:47.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:48.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:47.812+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:48.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:47.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.895578+0000) 2024-02-19T20:27:48.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:47.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:48.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:47.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:48.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:47.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:48.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:47.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:48.438 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:48.232+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.233989+0000) 2024-02-19T20:27:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:48.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:48.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:48.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:48.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:48.991 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:48.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.738296+0000) 2024-02-19T20:27:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:48.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:48.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:48.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:48.992 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:48.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.895717+0000) 2024-02-19T20:27:49.258 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:49.143+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63696 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:49.258 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:49.137+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.138964+0000) 2024-02-19T20:27:49.259 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:49.137+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.138734+0000) 2024-02-19T20:27:49.259 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:48.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:49.259 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:48.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:49.259 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:48.997+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:49.259 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:48.997+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:49.259 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:49.141+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.142964+0000) 2024-02-19T20:27:49.260 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:49.232+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.234177+0000) 2024-02-19T20:27:49.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:49.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:49.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:49.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:49.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:49.457+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:49.984 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:49.736+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.738435+0000) 2024-02-19T20:27:49.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:49.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:49.984 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:49.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:49.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:49.744+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:49.985 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:49.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.895819+0000) 2024-02-19T20:27:50.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:49.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:50.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:49.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:50.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:49.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:50.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:49.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:50.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:50.236+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.237648+0000) 2024-02-19T20:27:50.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:50.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:50.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:50.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:50.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:50.473+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:51.020 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:50.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.738537+0000) 2024-02-19T20:27:51.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:50.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:51.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:50.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:51.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:50.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:51.021 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:50.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.895929+0000) 2024-02-19T20:27:51.021 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:50.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:51.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:50.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:51.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:50.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:51.022 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:50.972+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:51.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:51.239+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.241120+0000) 2024-02-19T20:27:51.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:51.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:51.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:51.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:51.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:51.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:52.022 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:51.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.738647+0000) 2024-02-19T20:27:52.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:51.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:52.022 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:51.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:52.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:51.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:52.023 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:51.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.896085+0000) 2024-02-19T20:27:52.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:51.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:52.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:51.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:52.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:51.933+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:52.023 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:51.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:52.298 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:52.241+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.243191+0000) 2024-02-19T20:27:52.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:52.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:52.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:52.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:52.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:52.417+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:53.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:52.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.738791+0000) 2024-02-19T20:27:53.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:52.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:53.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:52.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:53.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:52.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:53.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:52.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.896271+0000) 2024-02-19T20:27:53.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:52.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:53.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:52.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:53.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:52.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:53.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:52.959+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:53.447 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:53.242+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.244100+0000) 2024-02-19T20:27:53.713 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:53.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:53.714 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:53.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:53.714 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:53.452+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:53.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:53.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.738931+0000) 2024-02-19T20:27:53.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:53.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:53.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:53.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:53.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:53.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:53.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:53.894+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.896420+0000) 2024-02-19T20:27:53.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:53.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:53.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:53.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:53.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:53.941+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:54.470 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:54.150+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63750 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:54.471 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:54.144+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.145809+0000) 2024-02-19T20:27:54.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:54.144+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.145873+0000) 2024-02-19T20:27:54.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:54.146+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.148345+0000) 2024-02-19T20:27:54.472 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:54.242+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.244380+0000) 2024-02-19T20:27:54.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:54.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:54.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:54.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:54.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:54.488+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:54.741 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:54.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.739091+0000) 2024-02-19T20:27:55.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:54.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:55.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:54.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:55.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:54.746+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:55.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:54.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.896478+0000) 2024-02-19T20:27:55.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:54.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:55.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:54.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:55.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:54.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:55.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:54.925+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:55.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:55.244+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.246384+0000) 2024-02-19T20:27:55.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:55.729+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:55.786 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:55.729+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:55.787 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:55.729+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:55.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:55.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:55.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:55.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:55.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:55.517+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:55.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:55.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.739232+0000) 2024-02-19T20:27:56.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:55.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.896554+0000) 2024-02-19T20:27:56.054 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:55.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:56.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:55.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:56.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:55.967+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:56.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:55.967+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:56.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:56.246+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.248402+0000) 2024-02-19T20:27:56.760 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:56.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:56.760 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:56.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:56.760 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:56.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:56.761 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:56.737+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.739384+0000) 2024-02-19T20:27:57.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:56.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:57.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:56.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:57.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:56.765+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:57.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:56.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.896639+0000) 2024-02-19T20:27:57.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:56.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:57.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:56.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:57.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:56.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:57.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:56.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:57.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:57.249+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.250505+0000) 2024-02-19T20:27:57.787 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:57.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:57.788 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:57.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:57.788 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:57.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:57.788 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:57.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.739507+0000) 2024-02-19T20:27:58.054 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:57.806+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:58.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:57.806+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:58.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:57.806+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:58.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:57.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.896748+0000) 2024-02-19T20:27:58.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:57.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:58.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:57.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:58.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:57.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:58.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:57.916+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:58.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:58.249+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.251407+0000) 2024-02-19T20:27:58.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:58.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:58.732 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:58.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:58.733 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:58.478+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:59.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:58.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.739655+0000) 2024-02-19T20:27:59.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:58.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:59.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:58.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:59.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:58.814+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:27:59.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:58.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.896883+0000) 2024-02-19T20:27:59.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:58.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:59.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:58.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:59.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:58.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:27:59.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:58.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:59.449 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:27:59.158+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63809 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:27:59.449 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:59.151+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.153370+0000) 2024-02-19T20:27:59.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:59.154+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.156137+0000) 2024-02-19T20:27:59.450 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:59.250+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.251579+0000) 2024-02-19T20:27:59.450 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:59.152+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.153457+0000) 2024-02-19T20:27:59.450 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:59.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:59.450 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:59.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:27:59.450 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:59.454+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:00.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:27:59.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.739782+0000) 2024-02-19T20:28:00.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:59.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:00.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:59.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:00.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:59.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:00.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:27:59.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.897033+0000) 2024-02-19T20:28:00.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:59.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:00.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:59.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:00.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:59.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:00.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:27:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:27:59.916+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:00.409 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:00.252+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.253726+0000) 2024-02-19T20:28:00.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:00.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:00.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:00.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:00.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:00.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:01.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:00.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.739879+0000) 2024-02-19T20:28:01.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:00.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:01.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:00.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:01.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:00.830+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:01.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:00.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.897180+0000) 2024-02-19T20:28:01.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:00.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:01.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:00.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:01.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:00.884+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:01.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:00.884+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:01.408 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:01.255+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.256855+0000) 2024-02-19T20:28:01.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:01.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:01.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:01.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:01.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:01.414+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:02.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:01.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.739973+0000) 2024-02-19T20:28:02.032 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:01.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:02.032 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:01.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:02.032 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:01.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:02.032 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:01.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:02.032 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:01.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:02.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:01.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:02.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:01.815+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:02.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:01.895+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.897345+0000) 2024-02-19T20:28:02.299 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:02.257+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.258886+0000) 2024-02-19T20:28:02.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:02.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:02.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:02.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:02.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:02.405+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:03.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:02.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.740082+0000) 2024-02-19T20:28:03.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:02.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:03.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:02.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:03.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:02.875+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:03.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:02.875+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:03.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:02.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:03.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:02.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:03.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:02.768+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:03.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:02.896+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.897481+0000) 2024-02-19T20:28:03.387 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:03.259+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.259880+0000) 2024-02-19T20:28:03.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:03.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:03.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:03.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:03.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:03.393+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:04.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:03.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.740194+0000) 2024-02-19T20:28:04.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:03.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:04.055 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:03.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:04.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:03.829+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:04.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:03.829+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:04.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:03.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:04.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:03.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:04.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:03.807+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:04.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:03.896+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.897569+0000) 2024-02-19T20:28:04.197 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T20:28:04.216 INFO:tasks.cephadm:Teardown begin 2024-02-19T20:28:04.216 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-02-19T20:28:04.257 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T20:28:04.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:28:04.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:28:04.345 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:04.165+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63861 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:04.346 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:04.159+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.160898+0000) 2024-02-19T20:28:04.346 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:04.163+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.164823+0000) 2024-02-19T20:28:04.346 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:04.258+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.260078+0000) 2024-02-19T20:28:04.347 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:04.159+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.161379+0000) 2024-02-19T20:28:04.347 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:04.351+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:04.347 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:04.351+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:04.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:04.351+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:05.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:04.738+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.740300+0000) 2024-02-19T20:28:05.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:04.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:05.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:04.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:05.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:04.787+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:05.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:04.896+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.897657+0000) 2024-02-19T20:28:05.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:04.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:05.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:04.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:05.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:04.827+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:05.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:04.827+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:05.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:05.262+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.263567+0000) 2024-02-19T20:28:05.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:05.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:05.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:05.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:05.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:05.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:06.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:05.739+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.740375+0000) 2024-02-19T20:28:06.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:05.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:06.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:05.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:06.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:05.764+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:06.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:05.897+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.897748+0000) 2024-02-19T20:28:06.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:05.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:06.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:05.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:06.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:05.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:06.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:05.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:06.328 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:06.265+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.265619+0000) 2024-02-19T20:28:06.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:06.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:06.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:06.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:06.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:06.335+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:07.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:06.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.740496+0000) 2024-02-19T20:28:07.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:07.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:07.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:06.747+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:07.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:06.897+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.897862+0000) 2024-02-19T20:28:07.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:06.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:07.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:06.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:07.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:06.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:07.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:06.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:07.356 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:28:07.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:07.267+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.267800+0000) 2024-02-19T20:28:07.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:07.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:07.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:07.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:07.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:07.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:08.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:07.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.740586+0000) 2024-02-19T20:28:08.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:07.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:08.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:07.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:08.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:07.776+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:08.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:07.897+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.897951+0000) 2024-02-19T20:28:08.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:07.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:08.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:07.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:08.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:07.913+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:08.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:07.913+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:08.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:08.268+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.268999+0000) 2024-02-19T20:28:08.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:08.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:08.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:08.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:08.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:08.313+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:09.055 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:08.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.740696+0000) 2024-02-19T20:28:09.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:08.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:09.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:08.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:09.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:08.756+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:09.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:08.897+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.898062+0000) 2024-02-19T20:28:09.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:08.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:09.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:08.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:09.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:08.899+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:09.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:08.899+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:09.307 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:09.172+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63914 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:09.308 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:09.167+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.167586+0000) 2024-02-19T20:28:09.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:09.167+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.168359+0000) 2024-02-19T20:28:09.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:09.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:09.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:09.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:09.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:09.170+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.170488+0000) 2024-02-19T20:28:09.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:09.268+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.269278+0000) 2024-02-19T20:28:10.056 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:09.314+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:10.056 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:09.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.740810+0000) 2024-02-19T20:28:10.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:09.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:10.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:09.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:10.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:09.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:10.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:09.898+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:10.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:09.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:10.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:09.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:10.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:09.758+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:10.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:09.897+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.898191+0000) 2024-02-19T20:28:10.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:10.270+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.271290+0000) 2024-02-19T20:28:10.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:10.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:10.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:10.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:10.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:10.272+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:10.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:10.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.740906+0000) 2024-02-19T20:28:10.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:10.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:10.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:10.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:10.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:10.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:10.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:10.897+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.898311+0000) 2024-02-19T20:28:10.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:10.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:10.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:10.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:10.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:10.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:10.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:10.933+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:11.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:11.273+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.273549+0000) 2024-02-19T20:28:11.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:11.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:11.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:11.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:11.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:11.278+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:11.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:11.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.741042+0000) 2024-02-19T20:28:11.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:11.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:11.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:11.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:11.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:11.695+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:11.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:11.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.898507+0000) 2024-02-19T20:28:11.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:11.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:11.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:11.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:11.971 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:11.909+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:11.972 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:11.909+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:12.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:12.275+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.275609+0000) 2024-02-19T20:28:12.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:12.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:12.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:12.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:12.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:12.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:12.799 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:12.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.741148+0000) 2024-02-19T20:28:12.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:12.729+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:12.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:12.729+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:12.799 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:12.729+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:13.056 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:12.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.898641+0000) 2024-02-19T20:28:13.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:12.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:13.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:12.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:13.056 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:12.941+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:13.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:12.941+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:13.333 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:13.276+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.276525+0000) 2024-02-19T20:28:13.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:13.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:13.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:13.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:13.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:13.340+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:14.056 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:13.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.741260+0000) 2024-02-19T20:28:14.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:13.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:14.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:13.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:14.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:13.964+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:14.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:13.965+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:14.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:13.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:14.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:13.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:14.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:13.762+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:14.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:13.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.898778+0000) 2024-02-19T20:28:14.321 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:14.179+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 63966 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:14.321 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:14.174+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.174731+0000) 2024-02-19T20:28:14.322 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:14.177+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.178167+0000) 2024-02-19T20:28:14.322 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:14.276+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.276808+0000) 2024-02-19T20:28:14.322 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:14.173+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.174433+0000) 2024-02-19T20:28:14.322 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:14.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:14.323 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:14.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:14.323 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:14.328+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:15.056 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:14.740+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.741371+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:14.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:14.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:14.774+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:14.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.898897+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:14.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:14.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:15.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:14.952+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:15.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:14.952+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:15.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:15.279+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.280264+0000) 2024-02-19T20:28:15.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:15.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:15.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:15.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:15.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:15.297+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:15.891 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:15.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.741470+0000) 2024-02-19T20:28:15.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:15.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:15.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:15.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:15.892 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:15.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:16.215 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:15.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.898994+0000) 2024-02-19T20:28:16.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:15.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:16.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:15.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:16.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:15.971+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:16.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:15.971+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:16.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:28:16.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:28:16.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:16.281+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.282392+0000) 2024-02-19T20:28:16.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:16.342+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:16.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:16.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:16.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:16.343+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:17.056 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:16.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.741570+0000) 2024-02-19T20:28:17.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:16.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:17.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:16.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:17.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:16.803+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:17.057 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:16.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.899082+0000) 2024-02-19T20:28:17.057 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:16.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:17.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:16.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:17.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:16.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:17.058 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:16.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:17.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:17.284+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.284494+0000) 2024-02-19T20:28:17.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:17.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:17.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:17.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:17.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:17.303+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:18.025 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:17.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.741659+0000) 2024-02-19T20:28:18.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:17.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:18.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:17.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:18.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:17.821+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:18.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:17.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.899187+0000) 2024-02-19T20:28:18.278 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:18.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:18.278 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:18.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:18.278 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:18.274+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:18.278 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:18.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:18.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:18.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:18.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:18.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:18.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:18.032+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:18.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:18.285+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.285505+0000) 2024-02-19T20:28:19.009 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:18.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.741768+0000) 2024-02-19T20:28:19.010 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:18.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:19.010 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:18.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:19.010 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:18.784+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:19.010 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:18.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.899298+0000) 2024-02-19T20:28:19.278 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:19.187+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64019 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:19.278 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:19.181+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.181499+0000) 2024-02-19T20:28:19.279 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:19.181+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.182246+0000) 2024-02-19T20:28:19.279 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:19.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:19.279 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:19.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:19.279 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:19.258+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:19.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:19.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:19.279 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:19.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:19.280 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:19.017+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:19.280 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:19.017+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:19.280 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:19.184+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.184956+0000) 2024-02-19T20:28:19.420 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:28:19.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:19.285+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.285683+0000) 2024-02-19T20:28:20.031 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:19.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.741879+0000) 2024-02-19T20:28:20.032 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:19.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:20.032 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:19.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:20.032 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:19.771+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:20.032 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:19.898+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.899423+0000) 2024-02-19T20:28:20.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:20.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:20.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:20.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:20.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:20.226+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:20.301 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:20.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:20.301 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:20.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:20.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:20.039+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:20.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:20.288+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.289251+0000) 2024-02-19T20:28:21.047 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:20.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.741980+0000) 2024-02-19T20:28:21.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:20.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:21.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:20.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:21.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:20.794+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:21.048 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:20.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.899503+0000) 2024-02-19T20:28:21.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:21.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:21.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:21.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:21.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:21.213+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:21.301 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:21.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:21.301 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:21.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:21.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:21.055+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:21.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:21.055+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:21.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:21.292+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.292747+0000) 2024-02-19T20:28:22.030 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:21.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.742096+0000) 2024-02-19T20:28:22.031 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:21.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:22.031 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:21.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:22.031 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:21.825+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:22.031 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:21.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.899603+0000) 2024-02-19T20:28:22.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:22.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:22.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:22.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:22.287 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:22.181+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:22.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:22.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:22.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:22.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:22.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:22.037+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:22.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:22.038+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:22.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:22.294+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.294798+0000) 2024-02-19T20:28:23.033 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:22.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.742209+0000) 2024-02-19T20:28:23.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:22.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:23.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:22.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:23.033 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:22.829+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:23.034 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:22.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.899691+0000) 2024-02-19T20:28:23.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:23.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:23.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:23.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:23.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:23.040+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:23.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:23.040+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:23.289 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:23.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:23.289 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:23.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:23.289 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:23.223+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:23.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:23.295+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.295915+0000) 2024-02-19T20:28:24.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:23.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.742329+0000) 2024-02-19T20:28:24.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:23.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:24.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:23.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:24.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:23.786+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:24.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:23.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.899781+0000) 2024-02-19T20:28:24.307 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:24.193+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64074 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:24.307 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:24.188+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.188739+0000) 2024-02-19T20:28:24.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:24.188+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.188689+0000) 2024-02-19T20:28:24.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:24.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:24.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:24.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:24.308 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:24.233+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:24.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:24.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:24.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:24.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:24.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:24.062+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:24.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:24.062+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:24.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:24.192+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.192532+0000) 2024-02-19T20:28:24.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:24.295+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.296139+0000) 2024-02-19T20:28:25.051 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:24.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.742470+0000) 2024-02-19T20:28:25.051 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:24.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:25.051 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:24.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:25.051 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:24.737+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:25.052 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:24.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.899869+0000) 2024-02-19T20:28:25.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:25.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:25.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:25.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:25.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:25.238+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:25.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:25.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:25.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:25.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:25.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:25.058+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:25.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:25.058+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:25.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:25.297+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.298239+0000) 2024-02-19T20:28:25.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:25.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.742561+0000) 2024-02-19T20:28:25.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:25.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:25.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:25.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:25.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:25.724+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:25.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:25.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.899962+0000) 2024-02-19T20:28:26.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:26.088+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:26.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:26.088+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:26.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:26.088+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:26.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:26.088+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:26.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:26.301+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.301799+0000) 2024-02-19T20:28:26.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:26.203+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:26.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:26.203+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:26.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:26.203+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:26.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:26.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.742699+0000) 2024-02-19T20:28:26.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:26.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:26.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:26.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:26.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:26.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:26.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:26.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.900102+0000) 2024-02-19T20:28:27.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:27.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:27.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:27.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:27.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:27.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:27.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:27.104+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:27.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:27.304+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.305287+0000) 2024-02-19T20:28:27.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:27.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:27.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:27.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:27.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:27.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:27.970 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:27.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.742807+0000) 2024-02-19T20:28:27.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:27.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:27.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:27.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:27.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:27.685+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:27.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:27.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.900248+0000) 2024-02-19T20:28:28.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:28.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:28.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:28.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:28.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:28.179+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:28.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:28.076+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:28.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:28.076+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:28.076+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:28.076+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:28.471 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:28.306+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.306481+0000) 2024-02-19T20:28:28.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:28.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.742902+0000) 2024-02-19T20:28:28.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:28.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:28.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:28.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:28.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:28.697+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:28.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:28.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.900439+0000) 2024-02-19T20:28:29.313 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:29.201+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64133 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:29.314 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:29.195+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.195987+0000) 2024-02-19T20:28:29.314 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:29.195+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.195765+0000) 2024-02-19T20:28:29.314 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:29.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:29.314 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:29.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:29.314 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:29.214+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:29.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:29.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:29.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:29.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:29.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:29.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:29.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:29.061+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:29.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:29.198+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.199205+0000) 2024-02-19T20:28:29.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:29.306+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.306754+0000) 2024-02-19T20:28:29.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:29.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.742997+0000) 2024-02-19T20:28:29.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:29.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:29.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:29.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:29.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:29.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:29.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:29.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.900570+0000) 2024-02-19T20:28:30.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:30.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:30.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:30.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:30.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:30.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:30.301 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:30.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:30.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.039+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:30.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.040+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:30.708 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.307+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.308775+0000) 2024-02-19T20:28:30.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:30.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.743080+0000) 2024-02-19T20:28:30.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:30.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:30.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:30.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:30.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:30.714+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:30.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:30.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.900665+0000) 2024-02-19T20:28:31.301 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:31.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:31.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:31.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:31.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:31.216+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:31.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:31.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:31.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:31.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:30.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:31.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:28:31.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:28:31.687 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:31.309+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.311425+0000) 2024-02-19T20:28:31.957 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:31.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.743215+0000) 2024-02-19T20:28:31.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:31.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:31.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:31.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:31.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:31.694+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:31.957 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:31.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.900826+0000) 2024-02-19T20:28:32.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:31.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:32.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:31.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:32.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:31.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:32.215 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:31.963+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:32.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:32.313+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.314912+0000) 2024-02-19T20:28:32.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:32.221+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:32.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:32.221+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:32.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:32.221+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:32.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:32.741+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.743348+0000) 2024-02-19T20:28:32.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:32.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:32.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:32.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:32.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:32.693+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:32.971 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:32.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.900980+0000) 2024-02-19T20:28:33.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:33.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:33.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:33.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:33.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:33.189+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:33.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:32.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:33.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:32.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:33.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:32.984+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:33.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:32.984+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:33.686 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:33.315+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.316455+0000) 2024-02-19T20:28:33.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:33.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.743474+0000) 2024-02-19T20:28:33.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:33.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:33.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:33.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:33.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:33.692+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:33.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:33.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.901105+0000) 2024-02-19T20:28:34.302 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:34.207+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64191 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:34.302 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:34.201+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.202856+0000) 2024-02-19T20:28:34.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:34.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:34.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:34.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:34.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:34.175+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:34.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:34.201+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.203033+0000) 2024-02-19T20:28:34.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:34.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:34.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:34.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:34.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.204+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.206420+0000) 2024-02-19T20:28:34.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:28:34.651 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.315+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.316611+0000) 2024-02-19T20:28:34.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:34.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.743583+0000) 2024-02-19T20:28:34.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:34.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:34.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:34.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:34.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:34.657+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:34.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:34.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.901214+0000) 2024-02-19T20:28:35.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:35.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:35.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:35.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:34.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:35.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:35.153+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:35.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:35.153+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:35.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:35.153+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:35.673 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:35.317+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.318689+0000) 2024-02-19T20:28:35.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:35.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.743678+0000) 2024-02-19T20:28:35.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:35.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:35.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:35.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:35.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:35.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:35.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:35.899+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.901334+0000) 2024-02-19T20:28:36.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:36.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:36.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:36.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:36.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:36.004+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:36.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:36.004+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:36.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:36.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:36.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:36.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:36.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:36.169+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:36.683 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:36.320+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.321550+0000) 2024-02-19T20:28:36.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:36.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.743795+0000) 2024-02-19T20:28:36.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:36.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:36.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:36.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:36.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:36.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:36.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:36.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.901462+0000) 2024-02-19T20:28:37.304 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:37.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:37.305 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:37.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:37.305 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:37.190+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:37.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:37.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:37.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:37.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:37.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:37.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:37.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:37.045+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:37.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:28:37.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:28:37.568 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:28:37.568 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:28:37.569 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T20:28:37.569 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T20:28:37.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:28:37.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:28:37.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:37.322+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.324062+0000) 2024-02-19T20:28:38.015 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:37.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.743904+0000) 2024-02-19T20:28:38.016 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:37.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:38.016 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:37.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:38.016 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:37.733+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:38.016 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:37.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.901577+0000) 2024-02-19T20:28:38.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:38.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:38.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:38.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:38.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:38.192+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:38.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:38.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:38.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:38.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:38.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:38.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:38.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:38.022+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:38.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:38.323+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.324974+0000) 2024-02-19T20:28:39.054 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:38.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.744022+0000) 2024-02-19T20:28:39.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:38.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:39.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:38.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:39.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:38.735+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:39.055 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:38.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.901716+0000) 2024-02-19T20:28:39.317 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:39.214+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64250 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:39.317 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:39.208+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.210120+0000) 2024-02-19T20:28:39.318 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:39.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:39.318 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:39.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:39.318 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:39.173+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:39.318 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:39.208+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.209962+0000) 2024-02-19T20:28:39.318 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:39.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:39.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:39.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:39.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:39.061+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:39.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:39.061+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:39.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:39.210+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.212406+0000) 2024-02-19T20:28:39.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:39.323+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.325210+0000) 2024-02-19T20:28:39.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:39.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.744142+0000) 2024-02-19T20:28:39.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:39.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:39.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:39.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:39.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:39.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:39.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:39.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.901877+0000) 2024-02-19T20:28:40.302 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:40.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:40.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:40.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:40.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:40.222+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:40.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:40.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:40.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:40.047+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:40.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:40.048+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:40.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:40.048+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:40.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:28:40.711 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:40.326+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.327632+0000) 2024-02-19T20:28:40.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:40.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.744239+0000) 2024-02-19T20:28:40.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:40.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:40.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:40.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:40.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:40.718+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:40.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:40.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.901933+0000) 2024-02-19T20:28:41.302 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:41.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:41.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:41.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:41.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:41.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:41.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:41.021+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:41.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:41.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:41.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:41.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:41.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:41.228+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:41.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:41.328+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.329885+0000) 2024-02-19T20:28:42.035 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:41.742+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.744351+0000) 2024-02-19T20:28:42.035 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:41.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:42.035 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:41.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:42.036 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:41.763+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:42.036 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:41.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.902028+0000) 2024-02-19T20:28:42.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:42.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:42.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:42.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:42.303 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:42.250+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:42.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:42.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:42.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:42.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:42.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:42.041+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:42.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:42.041+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:42.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:42.330+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.332083+0000) 2024-02-19T20:28:43.025 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:42.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.744467+0000) 2024-02-19T20:28:43.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:42.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:43.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:42.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:43.025 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:42.757+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:43.026 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:42.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.902137+0000) 2024-02-19T20:28:43.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:43.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:43.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:43.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:43.288 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:43.031+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:43.289 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:43.031+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:43.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:43.331+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.333008+0000) 2024-02-19T20:28:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:43.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:43.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:43.295+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:44.007 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:43.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.744564+0000) 2024-02-19T20:28:44.008 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:43.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:44.008 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:43.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:44.008 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:43.793+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:44.009 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:43.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.902248+0000) 2024-02-19T20:28:44.303 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:44.215+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.217349+0000) 2024-02-19T20:28:44.304 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:44.222+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64308 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:44.304 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:44.215+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.217054+0000) 2024-02-19T20:28:44.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:44.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:44.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:44.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:44.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:44.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:44.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:44.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:44.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:44.219+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.220493+0000) 2024-02-19T20:28:44.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:44.331+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.333188+0000) 2024-02-19T20:28:44.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:44.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:44.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:44.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:44.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:44.337+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:45.019 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:44.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.744672+0000) 2024-02-19T20:28:45.020 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:44.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:45.020 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:44.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:45.020 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:44.777+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:45.020 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:44.900+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.902409+0000) 2024-02-19T20:28:45.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:45.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:45.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:45.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.026+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:45.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.333+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.335183+0000) 2024-02-19T20:28:45.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:45.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:45.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:45.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:45.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:45.329+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:45.990 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:45.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.744783+0000) 2024-02-19T20:28:45.990 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:45.991 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:45.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:45.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:45.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:45.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:45.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:45.749+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:45.991 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:45.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.902544+0000) 2024-02-19T20:28:46.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.996+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:46.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:45.997+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:46.704 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:46.335+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.337217+0000) 2024-02-19T20:28:46.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:46.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:46.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:46.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:46.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:46.356+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:46.967 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:46.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.744906+0000) 2024-02-19T20:28:46.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:46.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:46.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:46.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:46.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:46.710+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:46.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:46.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.902632+0000) 2024-02-19T20:28:47.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:46.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:47.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:46.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:47.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:46.974+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:47.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:46.974+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:47.469 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:47.339+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.340660+0000) 2024-02-19T20:28:47.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:47.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:47.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:47.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:47.470 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:47.400+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:47.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:47.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.745015+0000) 2024-02-19T20:28:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:47.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:47.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:47.721+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:47.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:47.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.902746+0000) 2024-02-19T20:28:48.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:48.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:48.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:48.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:48.303 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:48.006+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:48.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:48.006+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:48.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:48.340+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.341631+0000) 2024-02-19T20:28:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:48.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:48.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:48.425+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:49.039 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:48.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.745130+0000) 2024-02-19T20:28:49.039 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:48.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:49.039 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:48.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:49.040 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:48.728+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:49.040 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:48.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.902902+0000) 2024-02-19T20:28:49.303 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:49.222+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.224416+0000) 2024-02-19T20:28:49.304 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:49.223+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.224859+0000) 2024-02-19T20:28:49.304 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:49.229+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64369 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:49.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:49.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:49.304 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:49.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:49.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:49.046+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:49.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:49.046+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:49.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:49.227+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.228758+0000) 2024-02-19T20:28:49.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:28:49.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:28:49.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:49.340+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.341837+0000) 2024-02-19T20:28:49.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:49.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:49.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:49.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:49.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:49.407+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:49.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:49.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.745227+0000) 2024-02-19T20:28:49.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:49.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:49.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:49.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:49.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:49.725+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:49.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:49.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.903007+0000) 2024-02-19T20:28:50.336 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:50.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:50.336 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:50.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:50.336 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:50.077+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:50.336 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:50.077+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:50.336 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:50.342+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.343881+0000) 2024-02-19T20:28:50.714 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:50.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:50.714 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:50.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:50.714 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:50.371+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:50.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:50.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.745309+0000) 2024-02-19T20:28:50.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:50.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:50.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:50.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:50.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:50.720+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:50.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:50.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.903104+0000) 2024-02-19T20:28:51.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:51.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:51.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:51.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:51.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:51.093+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:51.338 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:51.093+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:51.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:51.344+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.345951+0000) 2024-02-19T20:28:51.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:51.411+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:51.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:51.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:51.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:51.412+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:51.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:51.743+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.745425+0000) 2024-02-19T20:28:51.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:51.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:51.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:51.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:51.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:51.690+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:51.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:51.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.903218+0000) 2024-02-19T20:28:52.340 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:52.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:52.340 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:52.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:52.340 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:52.067+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:52.340 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:52.068+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:52.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:52.346+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.348225+0000) 2024-02-19T20:28:52.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:52.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:52.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:52.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:52.672 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:52.401+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:52.704 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:28:52.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:52.744+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.745522+0000) 2024-02-19T20:28:52.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:52.678+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:52.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:52.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:52.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:52.679+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:52.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:52.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.903309+0000) 2024-02-19T20:28:53.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:53.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:53.341 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:53.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:53.342 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:53.098+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:53.342 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:53.098+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:53.342 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:53.348+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.349625+0000) 2024-02-19T20:28:53.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:53.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:53.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:53.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:53.639 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:53.388+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:53.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:53.744+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.745669+0000) 2024-02-19T20:28:53.895 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:53.646+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:53.895 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:53.646+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:53.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:53.646+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:54.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:53.901+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.903408+0000) 2024-02-19T20:28:54.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:54.081+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:54.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:54.081+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:54.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:54.081+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:54.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:54.081+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:54.469 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:54.236+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64427 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:54.470 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:54.230+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.232061+0000) 2024-02-19T20:28:54.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:54.233+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.234734+0000) 2024-02-19T20:28:54.470 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:54.348+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.349840+0000) 2024-02-19T20:28:54.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:54.230+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.231944+0000) 2024-02-19T20:28:54.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:54.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:54.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:54.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:54.471 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:54.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:54.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:54.744+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.745761+0000) 2024-02-19T20:28:54.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:54.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:54.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:54.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:54.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:54.684+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:54.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:54.902+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.903507+0000) 2024-02-19T20:28:55.343 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:55.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:55.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:55.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:55.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:55.104+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:55.344 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:55.104+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:55.648 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:55.350+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.352017+0000) 2024-02-19T20:28:55.648 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:55.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:55.648 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:55.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:55.649 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:55.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:55.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:55.744+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.745861+0000) 2024-02-19T20:28:55.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:55.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:55.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:55.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:55.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:55.655+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:55.970 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:55.902+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.903590+0000) 2024-02-19T20:28:56.346 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:56.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:56.346 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:56.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:56.346 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:56.089+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:56.346 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:56.090+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:56.604 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:56.353+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.354141+0000) 2024-02-19T20:28:56.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:56.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:56.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:56.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:56.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:56.363+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:56.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:56.744+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.745974+0000) 2024-02-19T20:28:56.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:56.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:56.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:56.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:56.896 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:56.611+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:57.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:56.903+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.903732+0000) 2024-02-19T20:28:57.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:57.127+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:57.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:57.127+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:57.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:57.127+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:57.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:57.127+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:57.530 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:57.355+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.356607+0000) 2024-02-19T20:28:57.530 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:57.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:57.530 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:57.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:57.530 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:57.367+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:57.835 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:57.745+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.746125+0000) 2024-02-19T20:28:57.835 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:57.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:57.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:57.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:57.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:57.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:58.140 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:57.903+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.903900+0000) 2024-02-19T20:28:58.391 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:58.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:58.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:58.148+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:58.148+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:58.392 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:58.357+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.357590+0000) 2024-02-19T20:28:58.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:58.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:58.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:58.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:58.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:58.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:58.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:58.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:58.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:58.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:58.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:58.399+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:59.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:58.903+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.904073+0000) 2024-02-19T20:28:59.061 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:58.745+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.746307+0000) 2024-02-19T20:28:59.349 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:28:59.245+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64486 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:28:59.350 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:59.239+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.239482+0000) 2024-02-19T20:28:59.350 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:59.239+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.239871+0000) 2024-02-19T20:28:59.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:59.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:59.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:59.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:59.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:59.180+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:28:59.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:59.180+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:59.351 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:59.240+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.241263+0000) 2024-02-19T20:28:59.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:28:59.357+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.357802+0000) 2024-02-19T20:28:59.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:59.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:59.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:59.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:59.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:59.589+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:28:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:59.447+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:59.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:59.447+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:28:59.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:59.447+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:00.060 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:28:59.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.746493+0000) 2024-02-19T20:29:00.061 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:28:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:28:59.903+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.904251+0000) 2024-02-19T20:29:00.353 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:00.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:00.353 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:00.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:00.353 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:00.131+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:00.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:00.131+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:00.354 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:00.361+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.361478+0000) 2024-02-19T20:29:00.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:00.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:00.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:00.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:00.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:00.403+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:00.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:00.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:00.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:00.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:00.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:00.585+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:01.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:00.903+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.904404+0000) 2024-02-19T20:29:01.061 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:00.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.746616+0000) 2024-02-19T20:29:01.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:01.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:01.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:01.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:01.355 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:01.105+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:01.356 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:01.106+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:01.675 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:01.363+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.363710+0000) 2024-02-19T20:29:01.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:01.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:01.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:01.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:01.676 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:01.607+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:01.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:01.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:01.676 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:01.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:01.677 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:01.377+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:01.969 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:01.904+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.904607+0000) 2024-02-19T20:29:01.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:01.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.746757+0000) 2024-02-19T20:29:02.357 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:02.082+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:02.358 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:02.082+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:02.358 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:02.082+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:02.358 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:02.082+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:02.358 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:02.365+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.365903+0000) 2024-02-19T20:29:02.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:02.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:02.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:02.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:02.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:02.588+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:02.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:02.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:02.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:02.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:02.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:02.395+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:03.037 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:02.904+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.904726+0000) 2024-02-19T20:29:03.037 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:02.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.746844+0000) 2024-02-19T20:29:03.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:03.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:03.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:03.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:03.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:03.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:03.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:03.045+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:03.626 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:03.366+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.366911+0000) 2024-02-19T20:29:03.626 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:03.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:03.626 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:03.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:03.626 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:03.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:03.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:03.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.746940+0000) 2024-02-19T20:29:03.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:03.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:03.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:03.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:03.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:03.633+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:04.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:03.904+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.904835+0000) 2024-02-19T20:29:04.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:04.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:04.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:04.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:04.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:04.045+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:04.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:04.046+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:04.474 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:04.252+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64544 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:04.475 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:04.246+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.247067+0000) 2024-02-19T20:29:04.475 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:04.249+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.250338+0000) 2024-02-19T20:29:04.476 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:04.366+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.367144+0000) 2024-02-19T20:29:04.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:04.246+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.247154+0000) 2024-02-19T20:29:04.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:04.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:04.476 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:04.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:04.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:04.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:04.738 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:04.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:04.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:04.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:04.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:04.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:04.739 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:04.602+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:05.044 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:04.904+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.904973+0000) 2024-02-19T20:29:05.045 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:04.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.747058+0000) 2024-02-19T20:29:05.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:05.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:05.305 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:05.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:05.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:05.052+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:05.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:05.052+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:05.632 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:05.368+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.369094+0000) 2024-02-19T20:29:05.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:05.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:05.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:05.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:05.633 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:05.442+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:05.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:05.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.747193+0000) 2024-02-19T20:29:05.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:05.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:05.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:05.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:05.897 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:05.640+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:06.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:05.904+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.905094+0000) 2024-02-19T20:29:06.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:06.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:06.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:06.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:06.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:06.021+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:06.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:06.021+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:06.586 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:06.370+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.371223+0000) 2024-02-19T20:29:06.587 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:06.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:06.587 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:06.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:06.587 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:06.398+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:06.842 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:06.746+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.747363+0000) 2024-02-19T20:29:06.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:06.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:06.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:06.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:06.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:06.594+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:07.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:06.904+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.905270+0000) 2024-02-19T20:29:07.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:07.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:07.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:07.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:07.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:07.056+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:07.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:07.056+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:07.545 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:07.374+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.374686+0000) 2024-02-19T20:29:07.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:07.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:07.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:07.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:07.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:07.435+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:07.776 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:07.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:07.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.747495+0000) 2024-02-19T20:29:07.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:07.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:07.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:07.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:07.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:07.553+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:08.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:07.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.905486+0000) 2024-02-19T20:29:08.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:08.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:08.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:08.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:08.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:08.022+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:08.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:08.022+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:08.509 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:08.375+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.375695+0000) 2024-02-19T20:29:08.509 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:08.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:08.510 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:08.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:08.510 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:08.453+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:08.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:08.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.747625+0000) 2024-02-19T20:29:08.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:08.517+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:08.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:08.517+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:08.843 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:08.517+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:09.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:08.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.905635+0000) 2024-02-19T20:29:09.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:09.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:09.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:09.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:09.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:09.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:09.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:09.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:09.545 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:09.259+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64603 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:09.545 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:09.253+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.254195+0000) 2024-02-19T20:29:09.545 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:09.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:09.546 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:09.256+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.256889+0000) 2024-02-19T20:29:09.546 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:09.375+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.375899+0000) 2024-02-19T20:29:09.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:09.253+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.254039+0000) 2024-02-19T20:29:09.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:09.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:09.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:09.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:09.546 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:09.436+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:09.843 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:09.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.747725+0000) 2024-02-19T20:29:09.859 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:09.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:09.859 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:09.552+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:10.182 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:09.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.905750+0000) 2024-02-19T20:29:10.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:10.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:10.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.002+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:10.183 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.002+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:10.467 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.377+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.378087+0000) 2024-02-19T20:29:10.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:10.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:10.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:10.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:10.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:10.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:10.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:10.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:10.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:10.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:10.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:10.475+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:10.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:10.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:10.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:10.844 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:29:10.845 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T20:29:10.847 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631/remote/smithi039/crash 2024-02-19T20:29:10.848 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/crash -- . 2024-02-19T20:29:10.914 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631/remote/smithi086/crash 2024-02-19T20:29:10.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:10.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:11.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:10.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.747795+0000) 2024-02-19T20:29:11.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:10.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.905851+0000) 2024-02-19T20:29:11.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:11.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:11.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.987+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:11.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:10.987+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:11.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:11.379+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.380282+0000) 2024-02-19T20:29:11.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:11.519+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:11.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:11.519+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:11.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:11.519+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:11.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:11.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:11.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:11.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:11.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:11.510+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:12.061 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:11.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.905948+0000) 2024-02-19T20:29:12.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:11.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.747928+0000) 2024-02-19T20:29:12.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:11.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:12.062 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:11.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:12.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:11.949+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:12.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:11.949+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:12.647 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:12.382+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.382581+0000) 2024-02-19T20:29:12.648 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:12.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:12.648 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:12.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:12.648 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:12.497+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:12.648 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:12.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:12.649 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:12.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:12.649 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:12.506+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:12.897 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:12.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.748056+0000) 2024-02-19T20:29:13.219 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:12.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.906097+0000) 2024-02-19T20:29:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:12.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:12.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:12.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:13.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:12.959+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:13.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:13.383+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.383782+0000) 2024-02-19T20:29:13.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:13.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:13.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:13.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:13.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:13.476+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:13.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:13.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:13.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:13.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:13.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:13.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:13.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:14.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:13.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.906270+0000) 2024-02-19T20:29:14.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:13.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.748167+0000) 2024-02-19T20:29:14.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:13.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:14.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:13.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:14.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:13.982+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:14.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:13.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:14.375 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:14.266+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64661 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:14.376 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:14.260+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.261191+0000) 2024-02-19T20:29:14.376 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:14.263+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.263499+0000) 2024-02-19T20:29:14.376 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:14.260+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.260805+0000) 2024-02-19T20:29:14.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:14.383+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.383987+0000) 2024-02-19T20:29:14.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:14.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:14.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:14.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:14.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:14.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:14.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:14.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:14.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:14.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:14.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:14.496+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:15.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:14.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.906474+0000) 2024-02-19T20:29:15.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:14.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.748323+0000) 2024-02-19T20:29:15.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:14.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:15.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:14.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:15.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:14.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:15.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:14.986+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:15.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:15.385+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.386396+0000) 2024-02-19T20:29:15.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:15.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:15.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:15.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:15.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:15.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:15.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:15.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:15.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:15.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:15.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:15.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:16.005 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:15.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.748455+0000) 2024-02-19T20:29:16.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:15.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.906585+0000) 2024-02-19T20:29:16.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:16.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:16.306 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.013+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:16.307 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:16.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.388+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.388706+0000) 2024-02-19T20:29:16.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:16.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:16.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:16.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:16.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:16.486+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:16.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:16.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:16.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:16.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:16.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:16.468+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:17.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:16.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.748553+0000) 2024-02-19T20:29:17.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:16.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.906710+0000) 2024-02-19T20:29:17.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:17.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:17.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.963+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:17.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:16.963+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:17.685 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:17.390+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.390660+0000) 2024-02-19T20:29:17.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:17.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:17.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:17.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:17.686 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:17.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:17.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:17.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:17.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:17.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:17.687 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:17.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:17.940 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:17.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.906857+0000) 2024-02-19T20:29:17.941 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:17.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.748648+0000) 2024-02-19T20:29:18.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:17.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:18.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:17.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:18.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:17.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:18.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:17.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:18.487 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:18.391+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.391731+0000) 2024-02-19T20:29:18.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:18.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:18.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:18.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:18.740 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:18.495+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:18.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:18.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:18.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:18.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:18.741 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:18.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:19.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:18.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.906998+0000) 2024-02-19T20:29:19.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:18.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.748746+0000) 2024-02-19T20:29:19.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:18.923+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:19.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:18.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:19.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:18.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:19.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:18.924+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:19.383 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:19.267+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.267848+0000) 2024-02-19T20:29:19.383 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:19.273+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64720 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:19.384 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:19.270+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.271342+0000) 2024-02-19T20:29:19.384 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:19.267+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.267786+0000) 2024-02-19T20:29:19.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:19.391+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.391914+0000) 2024-02-19T20:29:19.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:19.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:19.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:19.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:19.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:19.524+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:19.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:19.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:19.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:19.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:19.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:19.498+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:20.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:19.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.748840+0000) 2024-02-19T20:29:20.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:19.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.907093+0000) 2024-02-19T20:29:20.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:19.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:20.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:19.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:20.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:19.887+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:20.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:19.887+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:20.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:20.393+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.394132+0000) 2024-02-19T20:29:20.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:20.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:20.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:20.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:20.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:20.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:20.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:20.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:20.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:20.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:20.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:20.451+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:21.062 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:20.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.907177+0000) 2024-02-19T20:29:21.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:20.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.748919+0000) 2024-02-19T20:29:21.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:20.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:21.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:20.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:21.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:20.837+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:21.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:20.838+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:21.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:21.396+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.397731+0000) 2024-02-19T20:29:21.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:21.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:21.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:21.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:21.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:21.437+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:21.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:21.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:21.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:21.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:21.723 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:21.574+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:22.062 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:21.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.749034+0000) 2024-02-19T20:29:22.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:21.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.907280+0000) 2024-02-19T20:29:22.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:21.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:22.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:21.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:22.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:21.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:22.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:21.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:22.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:22.398+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.399933+0000) 2024-02-19T20:29:22.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:22.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:22.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:22.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:22.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:22.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:22.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:22.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:22.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:22.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:22.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:22.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:22.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:22.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:23.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:22.905+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.907393+0000) 2024-02-19T20:29:23.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:22.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.749135+0000) 2024-02-19T20:29:23.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:22.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:23.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:22.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:23.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:22.869+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:23.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:22.869+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:23.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:23.399+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.401303+0000) 2024-02-19T20:29:23.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:23.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:23.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:23.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:23.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:23.452+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:23.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:23.504+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:23.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:23.504+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:23.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:23.504+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:24.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:23.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.907495+0000) 2024-02-19T20:29:24.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:23.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.749257+0000) 2024-02-19T20:29:24.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:23.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:24.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:23.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:24.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:23.870+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:24.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:23.870+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:24.392 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:24.279+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64778 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:24.392 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:24.273+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.274935+0000) 2024-02-19T20:29:24.393 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:24.275+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.277393+0000) 2024-02-19T20:29:24.393 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:24.273+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.274890+0000) 2024-02-19T20:29:24.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:24.400+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.401469+0000) 2024-02-19T20:29:24.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:24.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:24.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:24.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:24.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:24.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:24.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:24.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:24.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:24.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:24.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:24.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:25.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:24.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.907590+0000) 2024-02-19T20:29:25.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:24.747+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.749365+0000) 2024-02-19T20:29:25.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:24.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:25.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:24.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:25.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:24.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:25.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:24.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:25.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:25.402+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.403518+0000) 2024-02-19T20:29:25.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:25.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:25.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:25.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:25.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:25.438+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:25.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:25.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:25.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:25.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:25.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:25.494+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:26.048 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:26.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:25.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.907689+0000) 2024-02-19T20:29:26.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:25.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.749441+0000) 2024-02-19T20:29:26.063 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:25.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:26.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:25.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:26.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:25.917+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:26.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:25.918+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:26.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:26.404+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.405573+0000) 2024-02-19T20:29:26.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:26.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:26.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:26.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:26.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:26.445+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:26.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:26.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:26.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:26.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:26.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:26.535+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:27.063 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:26.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.749547+0000) 2024-02-19T20:29:27.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:26.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.907807+0000) 2024-02-19T20:29:27.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:26.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:27.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:26.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:27.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:26.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:27.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:26.954+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:27.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:27.406+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.407751+0000) 2024-02-19T20:29:27.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:27.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:27.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:27.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:27.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:27.439+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:27.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:27.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:27.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:27.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:27.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:27.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:28.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:27.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.907943+0000) 2024-02-19T20:29:28.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:27.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.749644+0000) 2024-02-19T20:29:28.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:27.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:28.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:27.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:28.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:27.912+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:28.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:27.912+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:28.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:28.407+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.408870+0000) 2024-02-19T20:29:28.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:28.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:28.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:28.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:28.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:28.518+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:28.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:28.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:28.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:28.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:28.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:28.455+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:29.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:28.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.908134+0000) 2024-02-19T20:29:29.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:28.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.749731+0000) 2024-02-19T20:29:29.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:28.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:29.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:28.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:29.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:28.898+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:29.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:28.898+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:29.399 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:29.286+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64837 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:29.399 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:29.280+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.281773+0000) 2024-02-19T20:29:29.400 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:29.283+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.284990+0000) 2024-02-19T20:29:29.400 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:29.280+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.281835+0000) 2024-02-19T20:29:29.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:29.407+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.409091+0000) 2024-02-19T20:29:29.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:29.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:29.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:29.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:29.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:29.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:29.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:29.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:29.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:29.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:29.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:29.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:30.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:29.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.908271+0000) 2024-02-19T20:29:30.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:29.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.749834+0000) 2024-02-19T20:29:30.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:29.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:30.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:29.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:30.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:29.939+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:30.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:29.939+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:30.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:30.409+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.411071+0000) 2024-02-19T20:29:30.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:30.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:30.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:30.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:30.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:30.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:30.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:30.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:30.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:30.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:30.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:30.505+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:31.063 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:30.906+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.908380+0000) 2024-02-19T20:29:31.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:30.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.749972+0000) 2024-02-19T20:29:31.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:30.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:31.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:30.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:31.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:30.983+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:31.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:30.983+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:31.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:31.411+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.413036+0000) 2024-02-19T20:29:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:31.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:31.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:31.454+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:31.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:31.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:31.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:31.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:31.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:31.508+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:32.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:31.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.908527+0000) 2024-02-19T20:29:32.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:31.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.750111+0000) 2024-02-19T20:29:32.064 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:31.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:32.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:31.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:32.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:31.940+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:32.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:31.940+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:32.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:32.413+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.415269+0000) 2024-02-19T20:29:32.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:32.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:32.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:32.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:32.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:32.515+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:32.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:32.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:32.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:32.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:32.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:32.504+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:33.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:32.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.908663+0000) 2024-02-19T20:29:33.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:32.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.750273+0000) 2024-02-19T20:29:33.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:32.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:33.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:32.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:33.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:32.890+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:33.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:32.890+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:33.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:33.414+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.416336+0000) 2024-02-19T20:29:33.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:33.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:33.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:33.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:33.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:33.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:33.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:33.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:33.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:33.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:33.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:33.484+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:34.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:33.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.908769+0000) 2024-02-19T20:29:34.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:33.748+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.750428+0000) 2024-02-19T20:29:34.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:33.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:34.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:33.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:34.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:33.934+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:34.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:33.934+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:34.404 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:34.291+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.292571+0000) 2024-02-19T20:29:34.405 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:34.294+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64895 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:34.405 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:34.288+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.289485+0000) 2024-02-19T20:29:34.406 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:34.287+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.289172+0000) 2024-02-19T20:29:34.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:34.415+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.416478+0000) 2024-02-19T20:29:34.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:34.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:34.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:34.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:34.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:34.505+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:34.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:34.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:34.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:34.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:34.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:34.541+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:35.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:34.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.908884+0000) 2024-02-19T20:29:35.064 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:34.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.750564+0000) 2024-02-19T20:29:35.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:34.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:35.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:34.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:35.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:34.925+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:35.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:34.925+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:35.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:35.416+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.418430+0000) 2024-02-19T20:29:35.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:35.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:35.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:35.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:35.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:35.481+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:35.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:35.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:35.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:35.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:35.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:35.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:36.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:35.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.909012+0000) 2024-02-19T20:29:36.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:35.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.750639+0000) 2024-02-19T20:29:36.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:35.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:36.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:35.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:36.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:35.930+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:36.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:35.930+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:36.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:36.419+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.420456+0000) 2024-02-19T20:29:36.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:36.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:36.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:36.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:36.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:36.474+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:36.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:36.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:36.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:36.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:36.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:36.475+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:37.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:36.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.909166+0000) 2024-02-19T20:29:37.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:36.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.750733+0000) 2024-02-19T20:29:37.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:36.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:37.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:36.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:37.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:36.948+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:37.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:36.948+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:37.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:37.421+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.422711+0000) 2024-02-19T20:29:37.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:37.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:37.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:37.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:37.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:37.488+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:37.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:37.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:37.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:37.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:37.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:37.493+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:38.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:38.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:38.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:37.907+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.909332+0000) 2024-02-19T20:29:38.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:37.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.750837+0000) 2024-02-19T20:29:38.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:37.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:38.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:37.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:38.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:37.986+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:38.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:37.987+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:38.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:38.422+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.423780+0000) 2024-02-19T20:29:38.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:38.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:38.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:38.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:38.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:38.472+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:38.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:38.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:38.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:38.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:38.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:38.498+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:39.064 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:38.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.909492+0000) 2024-02-19T20:29:39.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:38.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.750945+0000) 2024-02-19T20:29:39.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:38.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:39.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:38.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:39.065 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:38.985+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:39.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:38.985+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:39.414 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:39.301+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 64954 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:39.414 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:39.295+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.296526+0000) 2024-02-19T20:29:39.415 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:39.298+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.300242+0000) 2024-02-19T20:29:39.415 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:39.295+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.296472+0000) 2024-02-19T20:29:39.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:39.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:39.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:39.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:39.462 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:39.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:39.463 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:39.422+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.423958+0000) 2024-02-19T20:29:39.463 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:39.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:39.463 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:39.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:39.463 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:39.468+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:40.007 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:39.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.751087+0000) 2024-02-19T20:29:40.007 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:39.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.909637+0000) 2024-02-19T20:29:40.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:40.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:40.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:40.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:40.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:40.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:40.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:40.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:40.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:40.424+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.425937+0000) 2024-02-19T20:29:40.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:40.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:40.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:40.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:40.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:40.490+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:40.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:40.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:40.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:40.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:40.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:40.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:41.006 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:40.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.909763+0000) 2024-02-19T20:29:41.007 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:40.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.751198+0000) 2024-02-19T20:29:41.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:41.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:41.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:41.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.014+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:41.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.014+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:41.610 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.426+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.428110+0000) 2024-02-19T20:29:41.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:41.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:41.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:41.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:41.611 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:41.462+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:41.612 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:41.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:41.612 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:41.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:41.612 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:41.494+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:41.868 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:41.749+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.751374+0000) 2024-02-19T20:29:42.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:41.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.909914+0000) 2024-02-19T20:29:42.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:42.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:42.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.989+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:42.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:41.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:42.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:42.428+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.430120+0000) 2024-02-19T20:29:42.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:42.470+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:42.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:42.470+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:42.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:42.470+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:42.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:42.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:42.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:42.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:42.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:42.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:42.998 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:42.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.751565+0000) 2024-02-19T20:29:42.998 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:42.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.910002+0000) 2024-02-19T20:29:43.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:43.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:43.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:43.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:43.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:43.005+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:43.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:43.005+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:43.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:43.429+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.431261+0000) 2024-02-19T20:29:43.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:43.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:43.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:43.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:43.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:43.460+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:43.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:43.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:43.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:43.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:43.466+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:44.007 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:43.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.910102+0000) 2024-02-19T20:29:44.008 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:43.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.751698+0000) 2024-02-19T20:29:44.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:44.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:44.192 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:44.192 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:29:44.193 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T20:29:44.193 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T20:29:44.227 INFO:tasks.cephadm:Compressing logs... 2024-02-19T20:29:44.227 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T20:29:44.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:44.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:44.290 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:44.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:44.291 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:44.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:44.291 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:44.015+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:44.291 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:44.015+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:44.294 INFO:teuthology.orchestra.run.smithi039.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T20:29:44.295 INFO:teuthology.orchestra.run.smithi039.stderr:: No such file or directory 2024-02-19T20:29:44.295 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi039.omsmsr.log 2024-02-19T20:29:44.295 INFO:teuthology.orchestra.run.smithi039.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mon.smithi039.log 2024-02-19T20:29:44.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi039.omsmsr.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi039.aygwnv.log 2024-02-19T20:29:44.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mon.smithi039.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T20:29:44.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi039.aygwnv.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.1.log 2024-02-19T20:29:44.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-client.ceph-exporter.smithi039.log 2024-02-19T20:29:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.1.log: 91.2% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T20:29:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.0.log 2024-02-19T20:29:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-client.ceph-exporter.smithi039.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T20:29:44.309 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.log 2024-02-19T20:29:44.311 INFO:teuthology.orchestra.run.smithi039.stderr: 89.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T20:29:44.311 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.2.log 2024-02-19T20:29:44.311 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.log: 94.5% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-client.ceph-exporter.smithi039.log.gz 2024-02-19T20:29:44.311 INFO:teuthology.orchestra.run.smithi039.stderr: 85.7% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T20:29:44.312 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T20:29:44.314 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.2.log: 87.8% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T20:29:44.315 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mgr.smithi039.ydmpuh.log 2024-02-19T20:29:44.329 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-volume.log: /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mgr.smithi039.ydmpuh.log: 90.8% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T20:29:44.682 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:44.311+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65012 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:44.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:44.303+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.304540+0000) 2024-02-19T20:29:44.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:44.413+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:44.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:44.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:44.683 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:44.414+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:44.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:44.310+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.311708+0000) 2024-02-19T20:29:44.684 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:44.431+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.432469+0000) 2024-02-19T20:29:44.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:44.306+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.308050+0000) 2024-02-19T20:29:44.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:44.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:44.684 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:44.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:44.685 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:44.451+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:44.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:44.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.910210+0000) 2024-02-19T20:29:44.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:44.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.751830+0000) 2024-02-19T20:29:45.309 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:45.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:45.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.008+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:45.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.008+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:45.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:45.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:45.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:45.375+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:45.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:45.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:45.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:45.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:45.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:45.487+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:45.720 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.433+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.434988+0000) 2024-02-19T20:29:46.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:45.908+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.910311+0000) 2024-02-19T20:29:46.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:46.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:46.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:46.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:45.972+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:46.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:45.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.751930+0000) 2024-02-19T20:29:46.211 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mgr.smithi039.ydmpuh.log: file size changed while zipping 2024-02-19T20:29:46.211 INFO:teuthology.orchestra.run.smithi039.stderr: 89.8% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mgr.smithi039.ydmpuh.log.gz 2024-02-19T20:29:46.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:46.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:46.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:46.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:46.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:46.406+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:46.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:46.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:46.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:46.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:46.721 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:46.534+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:46.722 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:46.437+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.438805+0000) 2024-02-19T20:29:47.065 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:46.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.752065+0000) 2024-02-19T20:29:47.066 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:46.909+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.910438+0000) 2024-02-19T20:29:47.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:46.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:47.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:46.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:47.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:46.988+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:47.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:46.989+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:47.143 INFO:teuthology.orchestra.run.smithi039.stderr: 95.3% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi039.aygwnv.log.gz 2024-02-19T20:29:47.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mon.smithi039.log: file size changed while zipping 2024-02-19T20:29:47.353 INFO:teuthology.orchestra.run.smithi039.stderr: 93.7% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mon.smithi039.log.gz 2024-02-19T20:29:47.405 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-19T20:29:47.405 INFO:teuthology.orchestra.run.smithi039.stderr: 92.9% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.2.log.gz 2024-02-19T20:29:47.487 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-19T20:29:47.488 INFO:teuthology.orchestra.run.smithi039.stderr: 93.1% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.0.log.gz 2024-02-19T20:29:47.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:47.441+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.442623+0000) 2024-02-19T20:29:47.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:47.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:47.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:47.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:47.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:47.404+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:47.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:47.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:47.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:47.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:47.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:47.513+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-19T20:29:47.808 INFO:teuthology.orchestra.run.smithi039.stderr: 93.1% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-osd.1.log.gz 2024-02-19T20:29:48.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:47.909+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.910558+0000) 2024-02-19T20:29:48.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:47.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.752202+0000) 2024-02-19T20:29:48.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:47.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:48.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:47.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:48.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:47.942+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:48.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:47.942+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:48.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:48.442+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.443752+0000) 2024-02-19T20:29:48.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:48.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:48.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:48.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:48.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:48.385+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:48.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:48.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:48.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:48.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:48.554+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:49.065 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:48.909+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.910669+0000) 2024-02-19T20:29:49.066 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:48.750+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.752304+0000) 2024-02-19T20:29:49.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:48.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:49.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:48.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:49.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:48.960+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:49.066 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:48.960+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:49.317 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:49.319+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65074 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:49.317 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:49.313+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.314223+0000) 2024-02-19T20:29:49.317 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:49.313+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.314425+0000) 2024-02-19T20:29:49.318 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:49.316+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.316529+0000) 2024-02-19T20:29:49.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:49.443+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.443928+0000) 2024-02-19T20:29:49.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:49.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:49.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:49.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:49.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:49.566+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:49.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:49.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:49.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:49.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:49.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:49.364+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:50.002 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:49.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.910767+0000) 2024-02-19T20:29:50.002 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:49.751+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.752426+0000) 2024-02-19T20:29:50.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:50.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:50.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:50.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:50.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:50.010+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:50.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:50.010+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:50.603 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:50.446+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.446471+0000) 2024-02-19T20:29:50.604 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:50.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:50.604 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:50.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:50.608 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:50.360+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:50.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:50.611+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:50.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:50.611+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:50.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:50.611+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:50.879 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:50.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.752512+0000) 2024-02-19T20:29:51.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:50.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.910844+0000) 2024-02-19T20:29:51.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:51.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:51.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:51.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:51.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:51.025+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:51.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:51.025+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:51.554 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:51.449+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.449909+0000) 2024-02-19T20:29:51.555 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:51.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:51.555 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:51.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:51.555 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:51.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:51.563+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:51.563+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:51.563+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:51.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:51.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.752634+0000) 2024-02-19T20:29:52.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:51 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:51.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.910953+0000) 2024-02-19T20:29:52.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:52.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:52.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:52.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:52.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:52.029+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:52.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:52.029+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:52.559 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:52.451+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.451884+0000) 2024-02-19T20:29:52.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:52.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:52.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:52.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:52.560 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:52.350+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:52.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:52.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:52.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:52.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:52.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:52.568+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:52.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:52.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.752729+0000) 2024-02-19T20:29:53.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:52 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:52.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.911051+0000) 2024-02-19T20:29:53.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:53.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:53.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:53.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:53.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:53.012+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:53.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:53.013+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:53.579 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:53.452+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.453131+0000) 2024-02-19T20:29:53.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:53.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:53.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:53.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:53.580 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:53.329+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:53.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:53.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:53.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:53.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:53.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:53.588+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:53.851 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:53.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.752822+0000) 2024-02-19T20:29:54.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:53 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:53.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.911145+0000) 2024-02-19T20:29:54.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:54.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:54.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:54.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:54.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:54.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:54.220 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:54.060+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:54.539 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:54.327+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65132 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:54.539 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:54.323+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.324270+0000) 2024-02-19T20:29:54.540 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:54.452+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.453314+0000) 2024-02-19T20:29:54.540 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:54.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:54.540 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:54.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:54.540 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:54.318+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:54.540 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:54.321+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.321481+0000) 2024-02-19T20:29:54.541 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:54.321+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.321650+0000) 2024-02-19T20:29:54.541 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:54.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:54.541 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:54.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:54.541 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:54.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:54.880 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:54.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.752941+0000) 2024-02-19T20:29:55.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:54 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:54.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.911266+0000) 2024-02-19T20:29:55.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:55.050+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:55.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:55.050+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:55.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:55.050+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:55.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:55.050+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:55.522 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:55.455+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.456206+0000) 2024-02-19T20:29:55.522 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:55.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:55.522 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:55.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:55.522 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:55.334+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:55.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:55.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:55.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:55.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:55.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:55.530+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:55.883 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:55.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.753030+0000) 2024-02-19T20:29:56.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:55 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:55.910+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.911367+0000) 2024-02-19T20:29:56.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:56.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:56.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:56.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:56.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:56.026+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:56.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:56.027+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:56.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:29:56.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:29:56.535 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:56.457+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.458306+0000) 2024-02-19T20:29:56.536 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:56.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:56.536 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:56.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:56.536 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:56.369+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:56.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:56.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:56.803 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:56.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:56.804 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:56.544+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:56.804 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:56.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.753156+0000) 2024-02-19T20:29:57.066 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:56 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:56.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.911479+0000) 2024-02-19T20:29:57.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:57.050+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:57.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:57.050+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:57.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:57.050+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:57.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:57.050+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:57.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:57.324+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:57.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:57.324+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:57.451 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:57.324+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:57.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:57.460+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.460436+0000) 2024-02-19T20:29:57.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:57.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:57.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:57.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:57.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:57.516+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:58.005 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:57.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.911579+0000) 2024-02-19T20:29:58.006 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:57 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:57.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.753274+0000) 2024-02-19T20:29:58.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:58.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:58.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:58.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:58.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:58.060+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:58.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:58.060+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:58.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:58.461+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.461821+0000) 2024-02-19T20:29:58.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:58.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:58.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:58.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:58.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:58.322+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:58.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:58.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:58.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:58.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:58.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:58.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:59.066 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:58.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.911678+0000) 2024-02-19T20:29:59.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:58 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:58.752+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.753391+0000) 2024-02-19T20:29:59.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:59.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:59.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:59.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:59.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:59.080+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:29:59.320 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:59.080+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:59.388 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:29:59.718 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:29:59.336+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65191 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:29:59.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:59.332+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.333284+0000) 2024-02-19T20:29:59.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:29:59.461+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.462016+0000) 2024-02-19T20:29:59.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:59.330+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.330515+0000) 2024-02-19T20:29:59.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:59.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:59.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:59.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:59.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:59.352+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:29:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:59.329+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.329478+0000) 2024-02-19T20:29:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:59.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:59.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:29:59.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:59.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:00.058 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:29:59.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.911790+0000) 2024-02-19T20:30:00.058 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:29:59 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:29:59.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.753482+0000) 2024-02-19T20:30:00.310 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:00.066+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:00.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:00.066+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:00.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:00.066+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:00.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:00.067+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:00.571 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:00.463+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.464097+0000) 2024-02-19T20:30:00.571 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:00.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:00.571 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:00.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:00.572 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:00.393+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:00.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:00.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:00.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:00.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:00.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:00.580+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:00.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:00.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.753571+0000) 2024-02-19T20:30:01.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:00 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:00.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.911868+0000) 2024-02-19T20:30:01.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.033+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:01.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.033+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:01.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.033+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:01.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.033+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:01.522 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.465+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.466157+0000) 2024-02-19T20:30:01.523 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:01.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:01.523 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:01.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:01.523 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:01.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:01.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:01.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:01.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:01.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:01.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:01.531+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:01.889 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:01.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.753678+0000) 2024-02-19T20:30:02.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:01.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.911988+0000) 2024-02-19T20:30:02.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:02.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:02.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.992+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:02.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:01 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:01.993+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:02.488 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:02.467+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.468206+0000) 2024-02-19T20:30:02.489 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:02.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:02.489 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:02.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:02.489 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:02.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:02.645 INFO:teuthology.orchestra.run.smithi039.stderr: 92.9% -- replaced with /var/log/ceph/1026b4de-cf58-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi039.omsmsr.log.gz 2024-02-19T20:30:02.647 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-02-19T20:30:02.647 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m18.366s 2024-02-19T20:30:02.647 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m34.968s 2024-02-19T20:30:02.647 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.830s 2024-02-19T20:30:02.744 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:02.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:02.745 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:02.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:02.745 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:02.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:02.994 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:02.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.753785+0000) 2024-02-19T20:30:02.995 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:02 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:02.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.912083+0000) 2024-02-19T20:30:03.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:03.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:03.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.003+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:03.311 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.003+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:03.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.468+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.469402+0000) 2024-02-19T20:30:03.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:03.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:03.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:03.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:03.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:03.482+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:03.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:03.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:03.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:03.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:03.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:03.351+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:04.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:03.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.912194+0000) 2024-02-19T20:30:04.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:03.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.753888+0000) 2024-02-19T20:30:04.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:04.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:04.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.955+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:04.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:03 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:03.955+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:04.718 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:30:04.344+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65249 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:30:04.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:04.341+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.341739+0000) 2024-02-19T20:30:04.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:04.469+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.469667+0000) 2024-02-19T20:30:04.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:04.337+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.338217+0000) 2024-02-19T20:30:04.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:04.367+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:04.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:04.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:04.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:04.368+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:04.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:04.337+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.337931+0000) 2024-02-19T20:30:04.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:04.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:04.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:04.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:04.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:04.477+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:05.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:04.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.912303+0000) 2024-02-19T20:30:05.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:04.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.754003+0000) 2024-02-19T20:30:05.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:04.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:05.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:04.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:05.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:04.924+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:05.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:04 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:04.924+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:05.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:05.471+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.471715+0000) 2024-02-19T20:30:05.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:05.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:05.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:05.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:05.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:05.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:05.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:05.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:05.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:05.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:05.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:05.500+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:06.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:05.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.912398+0000) 2024-02-19T20:30:06.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:05.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.754096+0000) 2024-02-19T20:30:06.067 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:05.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:06.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:05.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:06.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:05.958+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:06.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:05 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:05.958+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:06.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:06.473+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.473647+0000) 2024-02-19T20:30:06.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:06.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:06.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:06.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:06.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:06.349+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:06.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:06.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:06.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:06.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:06.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:06.525+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:07.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:06.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.912484+0000) 2024-02-19T20:30:07.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:06.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.754207+0000) 2024-02-19T20:30:07.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:06.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:07.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:06.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:07.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:06.916+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:07.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:06 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:06.916+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:07.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:07.475+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.475678+0000) 2024-02-19T20:30:07.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:07.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:07.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:07.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:07.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:07.335+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:07.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:07.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:07.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:07.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:07.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:07.551+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:08.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:07.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.912588+0000) 2024-02-19T20:30:08.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:07.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.754313+0000) 2024-02-19T20:30:08.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:07.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:08.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:07.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:08.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:07.932+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:08.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:07 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:07.932+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:08.467 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:08.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:08.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:08.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:08.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:08.305+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:08.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:08.476+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.476768+0000) 2024-02-19T20:30:08.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:08.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:08.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:08.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:08.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:08.548+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:09.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:08.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.754421+0000) 2024-02-19T20:30:09.068 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:08.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.912698+0000) 2024-02-19T20:30:09.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:08.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:09.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:08.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:09.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:08.959+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:09.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:08 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:08.959+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:09.718 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:30:09.352+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65308 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:30:09.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:09.348+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.349360+0000) 2024-02-19T20:30:09.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:09.476+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.476993+0000) 2024-02-19T20:30:09.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:09.345+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.346290+0000) 2024-02-19T20:30:09.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:09.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:09.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:09.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:09.720 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:09.565+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:09.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:09.346+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.346915+0000) 2024-02-19T20:30:09.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:09.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:09.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:09.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:09.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:09.348+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:10.067 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:09.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.912842+0000) 2024-02-19T20:30:10.067 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:09.754+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.754543+0000) 2024-02-19T20:30:10.068 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:09.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:10.071 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:09.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:10.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:09.953+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:10.072 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:09 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:09.953+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:10.601 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:10.478+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.478975+0000) 2024-02-19T20:30:10.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:10.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:10.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:10.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:10.602 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:10.354+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:10.894 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:10.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:10.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:10.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:10.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:10.610+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:10.895 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:10.754+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.754630+0000) 2024-02-19T20:30:11.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:10.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.912921+0000) 2024-02-19T20:30:11.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:10.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:11.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:10.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:11.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:10.990+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:11.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:10 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:10.990+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:11.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:30:11.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:30:11.621 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:11.480+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.480977+0000) 2024-02-19T20:30:11.621 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:11.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:11.621 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:11.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:11.622 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:11.394+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:11.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:11.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:11.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:11.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:11.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:11.629+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:11.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:11.754+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.754721+0000) 2024-02-19T20:30:12.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:11 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:11.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.913014+0000) 2024-02-19T20:30:12.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:11.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:12.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:11.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:12.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:11.999+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:12.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:11.999+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:12.617 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:12.481+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.483001+0000) 2024-02-19T20:30:12.617 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:12.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:12.618 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:12.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:12.618 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:12.435+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:12.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:12.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:12.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:12.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:12.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:12.624+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:12.896 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:12.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.754811+0000) 2024-02-19T20:30:13.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:12.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.913122+0000) 2024-02-19T20:30:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:12.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:12.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:12.972+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:13.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:12 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:12.972+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:13.649 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:13.482+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.484134+0000) 2024-02-19T20:30:13.649 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:13.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:13.649 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:13.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:13.649 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:13.420+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:13.903 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:13.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:13.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:13.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:13.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:13.656+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:13.904 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:13.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.754933+0000) 2024-02-19T20:30:14.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:13.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.913229+0000) 2024-02-19T20:30:14.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:13.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:14.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:13.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:14.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:13.954+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:14.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:13 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:13.954+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:14.464 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:30:14.678 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:30:14.358+0000 7f0ff1007700 -1 mon.smithi039@0(probing) e3 get_health_metrics reporting 65365 slow ops, oldest is monmgrreport(gid 34104, 0 checks, 1 progress events) 2024-02-19T20:30:14.679 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:14.355+0000 7f43ba43a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.356669+0000) 2024-02-19T20:30:14.679 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:14.482+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.484360+0000) 2024-02-19T20:30:14.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:14.352+0000 7f8aac6d9700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.353823+0000) 2024-02-19T20:30:14.679 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:14.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:14.680 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:14.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:14.680 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:14.410+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:14.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:14.352+0000 7f259658c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.353774+0000) 2024-02-19T20:30:14.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:14.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:14.680 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:14.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:14.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:14.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.913323+0000) 2024-02-19T20:30:14.968 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:14.686+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:14.969 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:14.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.755087+0000) 2024-02-19T20:30:14.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:14.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:14.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:14.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:14.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:14.910+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:14.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:14 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:14.910+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:15.652 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:15.485+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.487011+0000) 2024-02-19T20:30:15.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:15.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:15.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:15.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:15.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:15.381+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:15.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:15.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:15.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:15.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:15.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:15.902+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:15.904 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:15.902+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:15.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:15.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:15.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:15.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:15.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:15.660+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:15.905 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:15.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.755175+0000) 2024-02-19T20:30:16.218 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:15 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:15.911+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.913399+0000) 2024-02-19T20:30:16.479 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:16.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:16.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:16.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:16.480 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:16.390+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:16.745 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:16.487+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.489157+0000) 2024-02-19T20:30:16.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:16.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:16.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:16.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:16.746 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:16.615+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:17.068 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:16.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.755293+0000) 2024-02-19T20:30:17.069 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:16.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.913483+0000) 2024-02-19T20:30:17.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:16.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:17.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:16.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:17.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:16.891+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:17.069 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:16 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:16.891+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:17.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:30:17.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:30:17.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:30:17.532 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:30:17.532 INFO:tasks.cephadm:Removing cluster... 2024-02-19T20:30:17.533 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1026b4de-cf58-11ee-95bb-87774f69a715 --force 2024-02-19T20:30:17.696 INFO:teuthology.orchestra.run.smithi039.stdout:Deleting cluster with fsid: 1026b4de-cf58-11ee-95bb-87774f69a715 2024-02-19T20:30:17.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:17.489+0000 7f43a9053700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.491199+0000) 2024-02-19T20:30:17.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:17.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:17.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:17.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:17.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:17.341+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:17.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:17.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:17.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:17.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:17.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:17.584+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:18.058 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:17 smithi039 systemd[1]: Stopping Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T20:30:18.059 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:17.753+0000 7f25891ad700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.755403+0000) 2024-02-19T20:30:18.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:17.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:18.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:17.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:18.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:17.867+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:18.059 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:17.867+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:18.060 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:17 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:17.912+0000 7f8a9b2f2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.913586+0000) 2024-02-19T20:30:18.468 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:30:18.066+0000 7f0ff400d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi039 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:30:18.468 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039[92220]: 2024-02-19T20:30:18.066+0000 7f0ff400d700 -1 mon.smithi039@0(probing) e3 *** Got Signal Terminated *** 2024-02-19T20:30:18.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:18.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:18.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:18.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:18.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:18.297+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:18.844 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:18 smithi039 podman[140694]: 2024-02-19 20:30:18.757876305 +0000 UTC m=+0.813399838 container died 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214) 2024-02-19T20:30:18.844 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:18.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:18.844 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:18.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:18.844 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:18.537+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:19.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:18.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:19.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:18.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:19.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:18.851+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:19.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:18 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:18.851+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:19.468 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:19 smithi039 podman[140694]: 2024-02-19 20:30:19.193782925 +0000 UTC m=+1.249306507 container cleanup 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:30:19.468 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:19 smithi039 bash[140694]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039 2024-02-19T20:30:19.468 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:19 smithi039 podman[140708]: 2024-02-19 20:30:19.403663626 +0000 UTC m=+0.649028907 container remove 3714643bde4a46dd66eefdc005f94e56529b768e058229b8ca466fb0f66d5250 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-mon-smithi039, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True) 2024-02-19T20:30:19.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:19.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:19.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:19.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:19.469 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:19.296+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:19.827 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:19 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service: Deactivated successfully. 2024-02-19T20:30:19.827 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:19 smithi039 systemd[1]: Stopped Ceph mon.smithi039 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T20:30:19.827 INFO:journalctl@ceph.mon.smithi039.smithi039.stdout:Feb 19 20:30:19 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@mon.smithi039.service: Consumed 3min 8.939s CPU time. 2024-02-19T20:30:19.827 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:19.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:19.828 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:19.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:19.828 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:19.547+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:20.108 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:19.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:20.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:19.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:20.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:19.845+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:20.109 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:19 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:19.845+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:20.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:20.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:20.416 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:20.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:20.417 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:20.339+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:20.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:20.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:20.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:20.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:20.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:20.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:21.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:20.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:21.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:20.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:21.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:20.871+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:21.133 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:20 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:20.872+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:21.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:21.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:21.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:21.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:21.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:21.326+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:21.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:21.514+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:21.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:21.514+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:21.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:21.514+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:22.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:21.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:22.112 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:21.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:22.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:21.883+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:22.113 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:21 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:21.883+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:22.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:22.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:22.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:22.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:22.459 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:22.343+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:22.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:22.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:22.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:22.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:22.710 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:22.492+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:22.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:22.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:22.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:22.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:22.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:22.895+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:22.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:22 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:22.895+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:23.603 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:23.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:23.603 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:23.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:23.603 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:23.319+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:23.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:23.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:23.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:23.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:23.604 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:23.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:24.154 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:23.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:24.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:23.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:24.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:23.874+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:24.155 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:23 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:23.874+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:24.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:24.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:24.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:24.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:24.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:24.353+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:24.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:24.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:24.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:24.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:24.888 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:24.499+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:25.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:24.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:25.184 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:24.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:25.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:24.906+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:25.185 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:24 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:24.907+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:25.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:25.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:25.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:25.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:25.460 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:25.386+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:25.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:25.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:25.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:25.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:25.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:25.479+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:26.129 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:25.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:26.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:25.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:26.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:25.951+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:26.130 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:25 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:25.951+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:26.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:26.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:26.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:26.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:26.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:26.433+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:26.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:26.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:26.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:26.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:26.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:26.481+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:27.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:26.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:27.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:26.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:27.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:26.904+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:27.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:26 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:26.904+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:27.652 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:27.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:27.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:27.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:27.653 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:27.428+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:27.653 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:27.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:27.653 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:27.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:27.653 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:27.443+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:27.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:27.919+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:27.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:27.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:27.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:27.920+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:27.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:27 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:27.920+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:28.704 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:28.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:28.704 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:28.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:28.704 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:28.422+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:28.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:28.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:28.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:28.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:28.705 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:28.460+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:28.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:28.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:28.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:28.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:28.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:28.876+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:28.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:28 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:28.876+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:29.383 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 systemd[1]: Stopping Ceph osd.0 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T20:30:29.384 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:29.373+0000 7f8aaa6d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:30:29.384 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:29.374+0000 7f8aaa6d5700 -1 osd.0 85 *** Got signal Terminated *** 2024-02-19T20:30:29.384 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:29.374+0000 7f8aaa6d5700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:30:29.384 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:29.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:29.384 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:29.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:29.384 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:29.391+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:29.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:29.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:29.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:29.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:29.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:29.502+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:30.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:29.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:30.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:29.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:30.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:29.849+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:30.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:29 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:29.849+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:30.489 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:30.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:30.490 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:30.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:30.490 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:30.380+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:30.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:30.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:30.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:30.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:30.796 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:30.497+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:31.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:30.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:31.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:30.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:31.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:30.805+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:31.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:30 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:30.805+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:31.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:31.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:31.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:31.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:31.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:31.459+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:31.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:31.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:31.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:31.400+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:32.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:31.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:32.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:31.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:32.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:31.810+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:32.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:31 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:31.810+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:32.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:32.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:32.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:32.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:32.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:32.415+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:32.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:32.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:32.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:32.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:32.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:32.412+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:33.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:32.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:33.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:32.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:33.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:32.814+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:33.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:32 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:32.814+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:33.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:33.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:33.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:33.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:33.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:33.406+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:33.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:33.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:33.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:33.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:33.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0[101297]: 2024-02-19T20:30:33.436+0000 7f8aa34c2700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:03.318275+0000 (oldest deadline 2024-02-19T20:26:23.318275+0000) 2024-02-19T20:30:34.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:33.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:34.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:33.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:34.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:33.832+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:34.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:33 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:33.832+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:34.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:34.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:34.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:34.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:34.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:34.389+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:35.217 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:34 smithi039 podman[141660]: 2024-02-19 20:30:34.977718442 +0000 UTC m=+5.708104224 container died 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-19T20:30:35.217 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:34.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:35.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:34.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:35.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:34.833+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:35.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:34 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:34.833+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:35.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:35 smithi039 podman[141660]: 2024-02-19 20:30:35.316707231 +0000 UTC m=+6.047093012 container cleanup 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-19T20:30:35.468 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:35 smithi039 bash[141660]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0 2024-02-19T20:30:35.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:35.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:35.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:35.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:35.469 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:35.381+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:35.828 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:35 smithi039 podman[141679]: 2024-02-19 20:30:35.520151812 +0000 UTC m=+0.538137394 container remove 601c2901cc7d5d3232d73711f881758be27ec222b53a4ce30060db70e2cbc5d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:30:36.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:35 smithi039 podman[141769]: 2024-02-19 20:30:35.8414326 +0000 UTC m=+0.018022971 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T20:30:36.089 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:35 smithi039 podman[141769]: 2024-02-19 20:30:35.970309507 +0000 UTC m=+0.146899923 container create 32121ce18ca86e4742781d5799b8b45a23c6dfe826b462dc865b26a74c56fc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:30:36.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:35.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:36.089 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:35.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:36.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:35.836+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:36.090 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:35 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:35.836+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:36.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:36.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:36.706 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:36.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:36.707 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:36.354+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:36.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:36 smithi039 podman[141769]: 2024-02-19 20:30:36.415572486 +0000 UTC m=+0.592162859 container init 32121ce18ca86e4742781d5799b8b45a23c6dfe826b462dc865b26a74c56fc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-02-19T20:30:36.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:36 smithi039 podman[141769]: 2024-02-19 20:30:36.418837369 +0000 UTC m=+0.595427742 container start 32121ce18ca86e4742781d5799b8b45a23c6dfe826b462dc865b26a74c56fc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:30:36.707 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:36 smithi039 podman[141769]: 2024-02-19 20:30:36.498306617 +0000 UTC m=+0.674897001 container attach 32121ce18ca86e4742781d5799b8b45a23c6dfe826b462dc865b26a74c56fc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:30:36.968 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:36 smithi039 podman[141769]: 2024-02-19 20:30:36.716191437 +0000 UTC m=+0.892781807 container died 32121ce18ca86e4742781d5799b8b45a23c6dfe826b462dc865b26a74c56fc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:30:36.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:36.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:36.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:36.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:36.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:36.815+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:36.969 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:36 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:36.815+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:37.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 systemd[1]: Stopping Ceph osd.1 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T20:30:37.718 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:37 smithi039 podman[141795]: 2024-02-19 20:30:37.293781349 +0000 UTC m=+0.568432291 container remove 32121ce18ca86e4742781d5799b8b45a23c6dfe826b462dc865b26a74c56fc47 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:30:37.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:37 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-19T20:30:37.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:37 smithi039 systemd[1]: Stopped Ceph osd.0 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T20:30:37.719 INFO:journalctl@ceph.osd.0.smithi039.stdout:Feb 19 20:30:37 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.0.service: Consumed 41.005s CPU time. 2024-02-19T20:30:37.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:37.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:37.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:37.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:37.719 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:37.318+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:38.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.771+0000 7f43b8436700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:30:38.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.771+0000 7f43b8436700 -1 osd.1 85 *** Got signal Terminated *** 2024-02-19T20:30:38.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.771+0000 7f43b8436700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:30:38.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:38.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:38.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.807+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:38.219 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:37 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:37.807+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:38.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:38.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:38.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:38.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:38.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:38.306+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:39.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:38.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:39.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:38.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:39.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:38.788+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:39.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:38 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:38.788+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:39.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:39.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:39.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:39.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:39.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:39.304+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:40.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:39.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:40.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:39.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:40.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:39.760+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:40.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:39 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:39.760+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:40.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:40.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:40.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:40.320+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:40.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:40.321+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:41.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:40.745+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:41.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:40.745+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:41.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:40.745+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:41.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:40 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:40.746+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:41.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:41.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:41.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:41.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:41.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:41.277+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:42.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:41.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:42.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:41.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:42.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:41.729+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:42.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:41 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:41.729+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:42.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:42.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:42.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:42.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:42.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:42.243+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:42.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:42.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:42.769+0000 7f43b1223700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:16.274365+0000 (oldest deadline 2024-02-19T20:26:36.274365+0000) 2024-02-19T20:30:43.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:42 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1[107160]: 2024-02-19T20:30:42.769+0000 7f43b1223700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.29:3925 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=43814b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:43.480 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:43 smithi039 podman[141883]: 2024-02-19 20:30:43.239262402 +0000 UTC m=+5.584711299 container died 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-19T20:30:43.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:43.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:43.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:43.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:43.481 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:43 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:43.271+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:43.897 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:43 smithi039 podman[141883]: 2024-02-19 20:30:43.589173766 +0000 UTC m=+5.934622668 container cleanup 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:30:43.898 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:43 smithi039 bash[141883]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1 2024-02-19T20:30:44.218 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:43 smithi039 podman[141902]: 2024-02-19 20:30:43.907184859 +0000 UTC m=+0.667808392 container remove 546fbde6ffab4e21a9628bb1544a61f236dbefa359fee5bcc11756c5ed8183eb (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-19T20:30:44.550 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:44.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:44.551 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:44.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:44.551 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:44 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:44.267+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:44.551 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:44 smithi039 podman[141992]: 2024-02-19 20:30:44.170476947 +0000 UTC m=+0.017166147 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T20:30:44.551 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:44 smithi039 podman[141992]: 2024-02-19 20:30:44.283202907 +0000 UTC m=+0.129892136 container create cd50ee23f5b3699823470c04751681198896bc6bdfa8fbb5f092eaab0b9fdd20 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:30:44.551 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:44 smithi039 podman[141992]: 2024-02-19 20:30:44.478154779 +0000 UTC m=+0.324843978 container init cd50ee23f5b3699823470c04751681198896bc6bdfa8fbb5f092eaab0b9fdd20 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-02-19T20:30:44.552 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:44 smithi039 podman[141992]: 2024-02-19 20:30:44.481563052 +0000 UTC m=+0.328252242 container start cd50ee23f5b3699823470c04751681198896bc6bdfa8fbb5f092eaab0b9fdd20 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:30:44.552 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:44 smithi039 podman[141992]: 2024-02-19 20:30:44.56042219 +0000 UTC m=+0.407111390 container attach cd50ee23f5b3699823470c04751681198896bc6bdfa8fbb5f092eaab0b9fdd20 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-19T20:30:44.968 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:44 smithi039 podman[142019]: 2024-02-19 20:30:44.804520408 +0000 UTC m=+0.018478476 container died cd50ee23f5b3699823470c04751681198896bc6bdfa8fbb5f092eaab0b9fdd20 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-19T20:30:45.315 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:45 smithi039 podman[142019]: 2024-02-19 20:30:45.305689061 +0000 UTC m=+0.519647128 container remove cd50ee23f5b3699823470c04751681198896bc6bdfa8fbb5f092eaab0b9fdd20 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-19T20:30:45.316 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:45.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:45.316 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:45.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:45.316 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:45.301+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:45.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 systemd[1]: Stopping Ceph osd.2 for 1026b4de-cf58-11ee-95bb-87774f69a715... 2024-02-19T20:30:45.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:45 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-19T20:30:45.718 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:45 smithi039 systemd[1]: Stopped Ceph osd.1 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T20:30:45.719 INFO:journalctl@ceph.osd.1.smithi039.stdout:Feb 19 20:30:45 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.1.service: Consumed 51.969s CPU time. 2024-02-19T20:30:46.218 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:45.775+0000 7f2594588700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:30:46.218 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:45.775+0000 7f2594588700 -1 osd.2 85 *** Got signal Terminated *** 2024-02-19T20:30:46.218 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:45 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:45.775+0000 7f2594588700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:30:46.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:46.260+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:46.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:46.260+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:46.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:46 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:46.260+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:47.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:47.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:47.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:47.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:47.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:47 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:47.284+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:48.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:48.264+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:48.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:48.264+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:48.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:48 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:48.264+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:49.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:49.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:49.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:49.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:49.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:49 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:49.308+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:50.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:50.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:50.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:50.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:50.718 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:50 smithi039 ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2[112726]: 2024-02-19T20:30:50.263+0000 7f259137d700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.86:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:26:05.869146+0000 (oldest deadline 2024-02-19T20:26:25.869146+0000) 2024-02-19T20:30:51.482 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:51 smithi039 podman[142097]: 2024-02-19 20:30:51.23103253 +0000 UTC m=+5.560634418 container died 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-19T20:30:51.849 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:51 smithi039 podman[142097]: 2024-02-19 20:30:51.600243405 +0000 UTC m=+5.929845290 container cleanup 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-19T20:30:51.850 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:51 smithi039 bash[142097]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2 2024-02-19T20:30:52.179 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:51 smithi039 podman[142114]: 2024-02-19 20:30:51.859639158 +0000 UTC m=+0.624333237 container remove 3d2286828d68c8429280c0a3fe5a8d21a52a35cce6ad56d0d45d5fcd9e0f163f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:30:52.436 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:52 smithi039 podman[142203]: 2024-02-19 20:30:52.091143091 +0000 UTC m=+0.017882248 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T20:30:52.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:52 smithi039 podman[142203]: 2024-02-19 20:30:52.193891846 +0000 UTC m=+0.120630999 container create 1e4cb1cc116604b184ebb84e9f7ea531e036babef0139caac27fd6b6627e911c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-19T20:30:52.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:52 smithi039 podman[142203]: 2024-02-19 20:30:52.380040989 +0000 UTC m=+0.306780146 container init 1e4cb1cc116604b184ebb84e9f7ea531e036babef0139caac27fd6b6627e911c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-19T20:30:52.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:52 smithi039 podman[142203]: 2024-02-19 20:30:52.383086909 +0000 UTC m=+0.309826065 container start 1e4cb1cc116604b184ebb84e9f7ea531e036babef0139caac27fd6b6627e911c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-19T20:30:52.437 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:52 smithi039 podman[142203]: 2024-02-19 20:30:52.446441062 +0000 UTC m=+0.373180225 container attach 1e4cb1cc116604b184ebb84e9f7ea531e036babef0139caac27fd6b6627e911c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:30:52.717 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:52 smithi039 podman[142203]: 2024-02-19 20:30:52.677142069 +0000 UTC m=+0.603881224 container died 1e4cb1cc116604b184ebb84e9f7ea531e036babef0139caac27fd6b6627e911c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:30:53.468 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:53 smithi039 podman[142230]: 2024-02-19 20:30:53.224586484 +0000 UTC m=+0.542220887 container remove 1e4cb1cc116604b184ebb84e9f7ea531e036babef0139caac27fd6b6627e911c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-1026b4de-cf58-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, ceph=True) 2024-02-19T20:30:53.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:53 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-19T20:30:53.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:53 smithi039 systemd[1]: Stopped Ceph osd.2 for 1026b4de-cf58-11ee-95bb-87774f69a715. 2024-02-19T20:30:53.472 INFO:journalctl@ceph.osd.2.smithi039.stdout:Feb 19 20:30:53 smithi039 systemd[1]: ceph-1026b4de-cf58-11ee-95bb-87774f69a715@osd.2.service: Consumed 37.106s CPU time. 2024-02-19T20:30:54.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:30:54.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:30:57.340 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:31:06.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:31:06.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:31:09.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:31:21.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:31:21.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:31:24.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:31:27.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:31:27.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:31:27.552 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:31:27.552 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:31:27.553 INFO:tasks.cephadm:Teardown complete 2024-02-19T20:31:27.553 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:31:27.579 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T20:31:27.591 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-02-19T20:31:27.608 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T20:31:27.608 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T20:31:27.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:31:27.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:31:30.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:31:39.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:31:39.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:31:42.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:31:54.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:31:54.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:31:57.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:00.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:00.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:00.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:00.892 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:32:00.949 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:32:00.972 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T20:32:00.985 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T20:32:00.986 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T20:32:00.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:00.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:01.009 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-02-19T20:32:01.015 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T20:32:01.015 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-02-19T20:32:01.015 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +560us[ +560us] +/- 78ms 2024-02-19T20:32:01.015 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -942us[ -973us] +/- 61ms 2024-02-19T20:32:01.015 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +808us[ +808us] +/- 85ms 2024-02-19T20:32:01.016 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:32:04.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:13.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:13.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:16.128 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:28.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:28.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:31.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:34.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:34.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:34.272 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:34.272 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:32:34.273 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:32:34.289 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T20:32:34.308 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T20:32:34.309 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T20:32:34.341 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T20:32:34.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:34.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:37.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:32:46.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:32:46.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:32:49.564 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:01.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:01.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:04.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:07.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:07.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:07.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:07.708 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:33:07.709 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:33:07.718 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T20:33:07.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T20:33:07.784 INFO:teuthology.task.internal:Duration was 6573.881091 seconds 2024-02-19T20:33:07.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T20:33:07.829 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T20:33:07.829 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T20:33:07.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:07.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:07.883 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T20:33:10.908 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:19.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:19.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:22.976 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:34.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:34.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:38.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:41.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:41.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:41.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:41.116 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:33:41.117 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:33:41.133 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T20:33:41.148 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T20:33:41.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T20:33:41.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:41.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:44.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:33:53.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:33:53.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:33:56.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:08.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:08.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:11.392 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:14.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:14.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:14.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:14.460 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:34:14.461 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:34:14.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T20:34:14.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T20:34:14.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:14.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:14.526 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-02-19T20:34:17.564 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:26.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:26.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:29.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:41.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:41.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:44.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:47.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:47.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:47.772 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:47.772 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:34:47.773 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:34:47.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T20:34:47.801 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T20:34:47.803 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631/remote/smithi039 2024-02-19T20:34:47.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T20:34:47.847 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631/remote/smithi086 2024-02-19T20:34:47.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:47.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:34:50.908 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:34:59.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:34:59.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:02.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:14.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:35:14.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:18.048 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:21.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:35:21.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:21.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:21.116 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:35:21.117 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:35:21.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T20:35:21.150 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T20:35:21.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T20:35:21.268 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T20:35:21.269 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T20:35:21.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:35:21.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:21.292 INFO:teuthology.orchestra.run.smithi039.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 19 20:31 /home/ubuntu/cephtest 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 19 20:34 /home/ubuntu/cephtest/archive 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 18:43 /home/ubuntu/cephtest/archive/syslog 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 265296 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 19 18:43 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 265295 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 19 18:43 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 530315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 20:32 /home/ubuntu/cephtest/archive/audit 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 530317 216 -rw------- 1 ubuntu ubuntu 218007 Feb 19 20:32 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-19T20:35:21.293 INFO:teuthology.orchestra.run.smithi039.stdout: 265381 212 -rwxr-xr-x 1 ubuntu ubuntu 214738 Feb 19 18:52 /home/ubuntu/cephtest/cephadm 2024-02-19T20:35:21.294 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-19T20:35:24.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:33.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:35:33.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:36.416 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:48.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:35:48.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:51.484 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:54.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-02-19T20:35:54.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:35:54.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-02-19T20:35:54.556 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:35:54.557 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 163, in umount self.client_remote.run(args=cmd, timeout=UMOUNT_TIMEOUT, omit_sudo=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 32: 'sudo umount /home/ubuntu/cephtest/mnt.1' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 121, in umount_all mount.umount() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/kernel_mount.py", line 166, in umount self.client_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-02-19T20:35:54.575 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T20:35:54.740 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T20:35:54.777 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: timeout: 2024-02-19T20:35:54.777 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T20:35:54.813 DEBUG:teuthology.nuke:shortname: smithi039 2024-02-19T20:35:54.813 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:35:54.833 DEBUG:teuthology.nuke:shortname: smithi086 2024-02-19T20:35:54.833 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:35:54.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:53.789618', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPe8hnfRIFRI8wD79N7u+QwiwW+B45WxJm/l+FtnC+lCvKXZUkLj9wLZNhUmYGPv/cKYsr8nvvvV06YhmdzkfHA='} 2024-02-19T20:35:54.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566631', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:53.791271', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSEW7KE6ueqFDGMgunBxOgX2IxzTCdBN4+AnkrFqCmo183I0NL8nT5Q+i1dipidEB4W7OyHGPOE2puDgLMta3Y='} 2024-02-19T20:35:54.892 INFO:teuthology.orchestra.console.smithi039:Power off 2024-02-19T20:35:54.892 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T20:35:54.917 DEBUG:teuthology.orchestra.console.smithi039:power off output: Chassis Power Control: Down/Off 2024-02-19T20:35:54.918 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:35:54.933 INFO:teuthology.orchestra.console.smithi086:Power off 2024-02-19T20:35:54.934 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T20:35:54.945 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-02-19T20:35:54.958 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-02-19T20:35:54.958 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:35:54.981 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-02-19T20:35:58.946 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:35:58.982 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:35:59.057 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-02-19T20:35:59.096 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-02-19T20:36:03.061 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:36:03.098 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:36:03.172 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-02-19T20:36:03.208 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-02-19T20:36:03.309 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-02-19T20:36:07.176 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:36:07.288 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is off 2024-02-19T20:36:07.389 INFO:teuthology.orchestra.console.smithi039:Power off completed 2024-02-19T20:36:07.495 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 6573.881090641022 failure_reason: 'Command failed (workunit test suites/fsstress.sh) on smithi039 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c97b0cb4c59b48a4866b8a2add1b65d9 status: fail success: false 2024-02-19T20:36:07.496 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T20:36:07.584 INFO:teuthology.run:FAIL